Extended reality control of systems and devices outside of a surgical sterile field

Abstract

A system for use in connection with a sterile field in a medical setting includes an external device outside of the sterile field and a mixed reality headset including a communication system in communicative connection with electronic circuitry thereof. The electronic circuitry is configured (i) to provide a mixed reality display via a display system, (ii) to store information regarding adjustment of settings for one or more parameters for control of the external device, (iii) to display a virtual interface for the settings through which the user can select values of the one or more parameters to define a state of the settings, and (iv) to transmit instruction data via the communication system of the state of the settings to an external system positioned external to the sterile field via which an adjustment of the one or more parameters may be effected to achieve the state of the settings.

Description

EXTENDED REALITY CONTROL OF SYSTEMS AND DEVICES OUTSIDE OF A SURGICAL STERILE FIELDCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims benefit of U.S. Provisional Patent Application Serial No. 63 / 710,865, filed October 23, 2024, the disclosure of which is incorporated herein by reference.BACKGROUND

[0002] The following information is provided to assist the reader in understanding technologies disclosed below and the environment in which such technologies may typically be used. The terms used herein are not intended to be limited to any particular narrow interpretation unless clearly stated otherwise in this document. References set forth herein may facilitate understanding of the technologies or the background thereof. The disclosure of all references cited herein are incorporated by reference.

[0003] A sterile field is a designated area within, for example, an operating room or a clinical setting that has been prepared to be essentially free of bacteria and particles or contaminants. A goal of creating a sterile field in, for example, the operating room is to prevent contamination of an open surgical wound via isolation of the operative side from a nonsterile surrounding environment. A surgical team creates and maintains a sterile field by following established principles as known in the medical arts. In general, all objects within the sterile field must be sterile, and sterile objects become unsterile when touched by unsterile objects. Sterile objects that are out of vision or below waist level may be considered unsterile.

[0004] Various devices, including surgical devices / machines, are used within the surgical field during surgical procedures. Associated devices or machines (for example, controllers for surgical devices used within the surgical field), such as the BOVIE® electrosurgical generator (available from Aspen Surgical of Caledonia, Michigan, USA), often have control interfaces that are not conducive to intraoperative adjustment because they must be positioned outside the sterile field of the operating room. If a device or system positioned external to the surgical field must be adjusted during surgery, manipulation is often performed by another member of the surgical team (for example, a technician positioned outside of the surgical field) through verbal instruction from an individual within the sterile field such as the surgeon and verbalconfirmation from the completion of the instructions by the member of the surgical team who manually adjusts the external device or system. Such a method of indirect adjustment results in extended procedure times, increase in the risk of confusion, and increase in the risk of error.SUMMARY

[0005] A system for use in connection with a sterile field in a medical setting includes an external device positioned outside of the sterile field and a mixed reality headset including a support system for supporting the mixed reality headset upon the head of a user who is located within the sterile field and electronic circuitry. The electronic circuitry includes a processor system and a memory system in communicative connection with the processor system. The memory system having stored therein one or more software algorithms executable by the processor system to control the mixed reality headset. The mixed reality headset further includes a display system attached to the support system and in communication with the electronic circuitry. The display system includes a display. The mixed reality headset further includes a communication system in communicative connection with the electronic circuitry. The electronic circuitry is configured (i) to provide a holographic or mixed reality display visible to the user via the display system, (ii) to store information regarding adjustment of settings, the settings being for one or more parameters for control of the external device, (iii) to display via the display a virtual interface for the settings through which the user can select values of the one or more parameters to define a state of the settings, and (iv) to transmit instruction data via the communication system of the state of the settings to an external system which is positioned external to the sterile field and via which an adjustment of the one or more parameters may be effected to achieve the state of the settings.

[0006] Use of the term “holographic display” is becoming less common in the spatial computer literature. As used herein, the term “holographic display” refers to a display in which virtual or digital information or objects are able to be situated or overlay ed upon the physical or real world (that is, displayed upon the physical or real world). Such displays are also sometimes referred to herein as mixed reality displays.

[0007] The external system may, for example, include at least one of (i) a communication system of an external communication device positioned external to the sterile field, the external communication device including a user interface to communicate the state of the settings to anexternal individual, who is located outside of the sterile field, to enable the external individual to physically manipulate controls of the external device to adjust the settings of the external device, (ii) a communication system of the external device, and (iii) a communication system of a system interface, the system interface being positioned outside of the sterile field, the system interface configured to physically manipulate controls of the external device to adjust the settings of the external device. In a number of embodiments, the system interface includes a robotic system. The external system may be configured to communicate information to the communication system of the mixed reality headset regarding a status of the adjustment of the one or more parameters.

[0008] In a number of embodiments, the external device is in connection with an internal device positioned within the sterile field and at least one of the one or more parameters includes a control parameter of the internal device. The internal device is a powered medical device such as a powered surgical device or a powered diagnostic device. The powered medical device may, for example, be a surgical drill device, a surgical saw device, an electrocautery device, laser emitting device, a microdebrider device, or an endoscope. In a number of embodiments, the powered medical device is an electrocautery device.

[0009] Communication between the mixed reality headset and the external system may, for example, occur via a communication hub.

[0010] The electronic circuitry may be further configured (i) to store instructional information which is different from the information regarding adjustment of settings, (ii) to display via the display a virtual interface for the instructional information via which the user can select parameters to define one or more instructions, and (iii) to transmit data via the communication system of the one or more instructions to the external system, wherein optionally the one or more instructions comprises information regarding surgical equipment to transport to the sterile field. The one or more instructions may be transmitted to the communication system of the external communication device.

[0011] A method for conducting a procedure in a sterile field in a medical setting includes providing a mixed reality headset including a support system for supporting the mixed reality headset upon the head of a user located within the sterile field, electronic circuitry including a processor system and a memory system in communicative connection with the processor system, the memory system having stored therein one or more software algorithms executableby the processor system to control the mixed reality headset, a display system attached to the support system and in communication with the electronic circuitry, the display system including a display, and a communication system in communicative connection with the electronic circuitry, the electronic circuitry being configured (i) to provide a holographic or mixed reality display visible to the user via the display system, (ii) to store information regarding adjustment of settings, the settings being for one or more parameters for control of the external device. The method further includes displaying via the display a virtual interface associated with the settings via which the user can select values of the one or more parameters to define a state of the settings, and transmitting instruction data via the communication system of the state of the settings to an external system via which an adjustment of the one or more parameters may be effected to achieve the state of the settings, the external system being positioned external to the sterile field.

[0012] The external system may, for example, include at least one of (i) a communication system of an external communication device positioned external to the sterile field, the external communication device including a user interface to communicate the state of the settings to an external individual, the external individual being located outside of the sterile field, to enable the external individual to physically manipulate controls of the external device to adjust the settings of the external device, (ii) a communication system of the external device, and (iii) a communication system of a system interface positioned outside of the sterile field, the system interface configured to physically manipulate controls of the external device to adjust the settings of the external device.

[0013] The electronic circuitry may further configured or operated (i) to store instructional information which is different from the information regarding adjustment of settings, (ii) to display via the display a virtual interface for the instructional information via which the user can select parameters to define one or more instructions, and (iii) to transmit data via the communication system of the one or more instructions to the external system, wherein optionally the one or more instructions comprises information regarding surgical equipment to transport to the sterile field. The one or more instructions may be transmitted to the communication system of the external communication device.

[0014] A system for use in connection with a sterile field in a medical setting includes an external device positioned outside of the sterile field and a mixed reality headset including a support system for supporting the mixed reality headset upon the head of a user who is locatedwithin the sterile field. The mixed reality headset also includes electronic circuitry including a processor system and a memory system in communicative connection with the processor system. The memory system has stored therein one or more software algorithms executable by the processor system to control the mixed reality headset. The mixed reality headset further includes a display system attached to the support system and in communication with the electronic circuitry. The display system includes a display (that is, a mixed reality display). The mixed reality headset further includes a communication system in communicative connection with the electronic circuitry. The electronic circuitry is configured (i) to provide a mixed reality display or image visible to the user via the display system, (ii) to store information regarding instructions to be sent to an external system which is positioned outside of the sterile field, (iii) to display via the display a virtual interface for information regarding the instructions via which the user can select one or more instructions, and (iv) to transmit the one or more instructions via the communication system to the external system.

[0015] Although particularly well suited for use in connection with a sterile field (for example, during surgery), devices, systems, and method hereof may be used in connection with any internal field or area to control an external device positioned outside of the internal field or area. In that regard, in another aspect a system hereof include an external device and a mixed reality headset including a support system for supporting the mixed reality headset system upon the head of a user who is located within the internal field. The mixed reality headset further includes electronic circuitry including a processor system and a memory system in communicative connection with the processor system. The memory system has stored therein one or more software algorithms executable by the processor system to control the mixed reality headset. The mixed reality headset further includes a display system attached to the support system and in communication with the electronic circuitry, wherein the display system includes a display, and a communication system in communicative connection with the electronic circuitry. The electronic circuitry is configured (i) to provide a holographic or mixed reality display visible to the user via the display system, (ii) to store information regarding adjustment of settings, the settings being for one or more parameters for control of the external device, (iii) to display via the display a virtual interface for the settings through which the user can select values of the one or more parameters to define a state of the settings, and (iv) to transmit instruction data via the communication system of the state of the settings to an external system which is positioned external to the internal field and via which an adjustment of the one or more parameters may be effected to achieve the state of the settings.

[0016] The external system may, for example, include at least one of (i) a communication system of an external communication device positioned external to the internal field, the external communication device including a user interface to communicate the state of the settings to an external individual, the external individual being located outside of the internal field, to enable the external individual to physically manipulate controls of the external device to adjust the settings of the external device, (ii) a communication system of the external device, and (iii) a communication system of a system interface positioned outside of the internal field, the system interface configured to physically manipulate controls of the external device to adjust the settings of the external device.

[0017] A mixed reality headset for use in an internal field or area includes a support system for supporting the mixed reality headset upon the head of a user who is located within the internal field and electronic circuitry including a processor system and a memory system in communicative connection with the processor system. The memory system has stored therein one or more software algorithms executable by the processor system to control the mixed reality headset. The mixed reality system further includes a display system attached to the support system and in communication with the electronic circuitry, wherein the display system includes a display, and a communication system in communicative connection with the electronic circuitry. The electronic circuitry is configured (i) to provide a holographic or mixed reality display visible to the user via the display system, (ii) to store information regarding adjustment of settings, the settings being for one or more parameters for control of an external device positioned outside of the internal field, (iii) to display via the display a virtual interface for the settings through which the user can select values of the one or more parameters to define a state of the settings, and (iv) to transmit instruction data via the communication system of the state of the settings to an external system which is positioned external to the internal field and via which an adjustment of the one or more parameters may be effected to achieve the state of the settings. The internal field may be a sterile field.

[0018] The present devices, systems, and methods, along with the attributes and attendant advantages thereof, will best be appreciated and understood in view of the following detailed description taken in conjunction with the accompanying drawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0019] FIG. 1A illustrates schematically an embodiment of a system hereof.

[0020] FIG. IB illustrates a mixed reality headset for use in the system of FIG. 1 A.

[0021] FIG. 1C illustrates another mixed reality headset for use in the system of FIG. 1 A.

[0022] FIG. 1C illustrates a representative embodiment of a holographic or mixed reality interface display for use in controlling a system or device outside or external to the sterile field.

[0023] FIG. ID illustrates another view of the holographic or mixed reality interface display of FIG. 1C.

[0024] FIG. 2 illustrates an embodiment of a flow diagram of actions of a system hereof.DETAILED DESCRIPTION

[0025] It will be readily understood that the components of the embodiments, as generally described and illustrated in the figures herein, may be arranged and designed in a wide variety of different configurations in addition to the described representative embodiments. Thus, the following more detailed description of the representative embodiments, as illustrated in the figures, is not intended to limit the scope of the embodiments, as claimed, but is merely illustrative of representative embodiments.

[0026] Reference throughout this specification to “one embodiment” or “an embodiment” (or the like) means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the appearance of the phrases “in one embodiment” or “in an embodiment” or the like in various places throughout this specification are not necessarily all referring to the same embodiment.

[0027] Furthermore, described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments. One skilled in the relevant art will recognize, however, that the various embodiments can be practiced without one or more of the specific details, or with other methods, components, materials, et cetera. Inother instances, well known structures, materials, or operations are not shown or described in detail to avoid obfuscation.

[0028] As used herein and in the appended claims, the singular forms “a,” “an,” and “the” include plural references unless the context clearly dictates otherwise. Thus, for example, reference to “a menu” includes a plurality of such menus and equivalents thereof known to those skilled in the art, and so forth, and reference to “the menu” is a reference to one or more such menus and equivalents thereof known to those skilled in the art, and so forth. Recitation of ranges of values herein are merely intended to serve as a shorthand method of referring individually to each separate value falling within the range. Unless otherwise indicated herein, each separate value, as well as intermediate ranges, are incorporated into the specification as if individually recited herein. All methods described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contraindicated by the text.

[0029] The terms “electronic circuitry,” “circuitry” or “circuit,” as used herein include, but is not limited to, hardware, firmware, software, or combinations of each to perform a function(s) or an action(s). For example, based on a desired feature or need, a circuit may include a software-controlled microprocessor, discrete logic such as an application specific integrated circuit (ASIC), or other programmed logic device. A circuit may also be fully embodied as software. As used herein, “circuit” is considered synonymous with “logic.” The term “logic,” as used herein includes, but is not limited to, hardware, firmware, software, or combinations of each to perform a function(s) or an action(s), or to cause a function or action from another component. For example, based on a desired application or need, logic may include a software-controlled microprocessor, discrete logic such as an application specific integrated circuit (ASIC), or other programmed logic device. Logic may also be fully embodied as software.

[0030] The term “processor,” as used herein includes, but is not limited to, one or more of virtually any number of processor systems or stand-alone processors, such as microprocessors, microcontrollers, central processing units (CPUs), and digital signal processors (DSPs), in any combination. The processor may be associated with various other circuits that support operation of the processor, such as random-access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read only memory (EPROM), clocks, decoders, memory controllers, or interrupt controllers, etc. These support circuits may be internal or external to the processor or its associated electronic packaging. Thesupport circuits are in operative communication with the processor. The support circuits are not necessarily shown separate from the processor in block diagrams or other drawings.

[0031] The term “memory system” refers to a collection of electronic components that store data and instructions. In computerized systems, a processor system can quickly access information stored in a memory system. Memory allows storage and retrieval of information and may, for example, include primary memory and secondary memory. Primary memory includes, for example, RAM, cache memory, etc. Secondary memory includes, for example, hard drives, hard disk drives etc.

[0032] The term “controller,” as used herein includes, but is not limited to, any circuit or device that coordinates and controls the operation of one or more input and / or output devices. A controller may, for example, include a device having one or more processors, microprocessors, or central processing units capable of being programmed to perform functions.

[0033] The term “software,” as used herein includes, but is not limited to, one or more computer readable or executable instructions that cause a computer or other electronic device to perform functions, actions, or behave in a desired manner. The instructions may be embodied in various forms such as routines, algorithms, modules, or programs including separate applications or code from dynamically linked libraries. Software may also be implemented in various forms such as a stand-alone program, a function call, a servlet, an applet, instructions stored in a memory, part of an operating system or other type of executable instructions. It will be appreciated by one of ordinary skill in the art that the form of software is dependent on, for example, requirements of a desired application, the environment it runs on, or the desires of a designer / programmer or the like.

[0034] As used herein, the term “approximately” when used in connection with a value refers to within 10% of the stated value, within 5% of the stated value, or within 1% of the stated value. As used herein the term “and / or” means one of or both of an entity. Thus, A and / or B means A or B, or both A and B.

[0035] In a number of embodiments hereof, a mixed reality system is used to communicate information or data between a user of the mixed reality system within the sterile field to a device outside or external to the sterile filed. Such information or data may be used to changeor adjust settings of certain devices (such as surgical or other device controllers), which are positioned outside the sterile field (for example, in an operating room setting). A mixed reality interface may be provided via the mixed reality system to control such devices outside the sterile filed by the person (that is, a user of the mixed reality system) within the sterile field without the need for oral communication with a person outside the sterile field. Devices which are outside of the sterile field are sometimes referenced herein as external devices. In a number of embodiments, the systems hereof may facilitate such adjustment while maintaining a specific focus on security and fault tolerance. Systems hereof may, for example, leverage advances in both Internet of Things (loT) and mixed reality (MR).

[0036] FIG. 1A illustrates an embodiment of a system 10 hereof. System 10 includes a mixed reality headset 100. Adaptation of augmented / mixed reality headsets for use in the medical field (for example, in the operating room) is disclosed in PCT International Publication Number WO 2024 / 077077, the disclosure of which is incorporated herein by reference. Such augmented / mixed reality headset systems, which include headsets having see-through displays, may, for example, be adapted from the MICROSOFT® HOLOLENS® 2 AR headsets available from Microsoft, Corporation of Redmond, WA or the MAGIC LEAP® 2 AR headset available from Magic Leap of Plantation, Florida. Augmented / mixed reality headsets suitable for use herein also include pass-through headsets. In pass-through headsets, the headset includes one or more external cameras or image sensors to provide an image or live video feed of one’s physical surroundings. Virtual objects / digital information is overlain upon the image / video feed. A representative example of a pass-through heads set suitable or adaptable for use herein is the APPLE VISION PRO headset available from Apple, Inc. of Cupertino, California. In pass-through displays, image processing and graphics rendering blend and display physical and generated reality in real time. Augmented reality (AR) and mixed reality (MR) are each technologies that overlay virtual or digital elements on the real world. MR further allows the physical and digital elements to interact with each other.

[0037] FIG. IB illustrates schematically an embodiment of headset 100, which includes a visor / display 110 (for example, a see-through display), which may, for example, be a rotatable (or pivotable) and see-through or pass-through display. A support, band, or arm system 120 including arms or bands 122, which connect to a rear pad 124, may be used to retain augmented reality headset 100 on the head of a user. An upper strap (not shown) may also be used to cooperate with the top of a user’s head. A communication system 130 may, for example, beprovided in a rear component housing 140. Communication system 130 may, for example, provide for wireless communication (for example, via BLUETOOTH, a wireless communication protocol administered by Bluetooth SIG, Inc. of Kirland, Washington, Wi-Fi, etc.) and wired communication (for example, via a universal serial bus or USB port, via ethernet, etc.). Headset system electronic circuitry 105 (electronics / components; illustrated schematically in FIG. IB) may, for example, be distributed between a front component housing / support 150 and rear component housing 140. As known in the art, an external power source (not illustrated) may be used to charge / power headset 100 (which typically includes a rechargeable battery system) through its native power input. Among other components electronic circuitry 105 may, for example, include a processing system and a memory system in communication with the processor system. Software executable by the processor system is stored in the memory system. In the case of a pass-through virtual reality display, one or more cameras 107 may be in communicative connection with the electronic circuitry to provide a live video feed of one’s physical surroundings. FIG. 1C illustrates schematically another embodiment of a headset 100a including a mixed reality display 110a (for example, a pass-through display) and a support system 120a to retain headset 100a in connection with a user’s head.

[0038] In FIG. 1 A, a user (for example, a physician / surgeon) is illustrated wearing headset 100 and holding powered surgical device such as an electrosurgical or electrocautery device 200 as known in the medical arts. Although a single surgical device 200 is illustrated in the representative example of FIG. 1 A, as described further below, a plurality of powered medical devices may be under the control of one or more surgeons or other medical staff within the surgical field during the course of a surgical procedure. In the illustrated embodiment, surgical device 200 includes a body 210 which is used as a handle by the user. As known in the art, one or more actuators 212 (for example, buttons, switches, etc.) may be included on body 210 which may be held by a user. Such actuators 212 may provide manual control functionality for a surgeon to manually control power / current to a tissue-interacting element 220, which may be an electrocautery probe, electrode, or knife positioned on a distal end of electrocautery surgical device 200. Connecting circuitry / wiring 240 is provided (illustrated as extending from a distal end of body 210 of electrocautery device 200) to connect electrocautery surgical device 200 to an external device 300 which functions as control device or controller / current generator. In current practice, powered surgical instruments or devices such as surgical device 200 may be connected directly to such an external device or controller including, for example, a powersource / current-generating system. As illustrated in FIG. 1A, a foot switch system 250 (positioned outside the sterile field) may be provided intermediate between surgical device 200 and external device 300. Actuators 212 (for example, buttons, etc.) and foot switch 250 provide limited regulation of, for example, the activation and intensity of a surgical device such as surgical device 200.

[0039] In a number of embodiments, system 10 also includes a communication hub 500, which includes electronic circuitry 510 including or in communicative connection with a communication system 520. In other embodiments, the functionality of the communication hub 500 as described herein may, for example, be incorporated into headset 100. Communication system 520 may, for example, provide for wireless communication (for example, via BLUETOOTH, Wi-Fi, etc.) and / or wired communication (for example, via USB, ethernet etc.). Communication hub 500 may coordinate communication between headset 100 and external communication systems to, for example, effect an adjustment of one or more parameters of an external device such as external device 300 (or first external device 300) through external device 300n (or nthexternal device 300n), as illustrated in FIG. 1 A. Each of external devices 300 through 300n may, for example, include electronic circuitry 310, 310n including a communication device or system 320, 320n.

[0040] Various external devices may, for example, be loT-enabled devices. Such devices can be controlled via communication of information / instructions directly from headset 100 or via communication hub 500. Because of safety and security concerns, however, a number of external devices such as external 300 may provide for only limited or no possibility of control thereof via communication from a remote device (such as headset 100 and / or communication hub 500). Such communication limitations are often present in the case of devices such as external device 300 which operate as a controller for surgical devices or other devices which come into contact with the patient.

[0041] In a number of embodiments, headset 100 hereof constructs a virtual user interface that can be displayed to the user intraoperatively via, for example, a holographic or mixed reality interface / display 1000 including one or more holographic or mixed reality menus 1010 via which, for example, a virtual interface 1010a (see FIG. 1 A) for a particular external device may be selected. During surgery, the physician may interact with the virtual user interface 1010a via several modalities, including but not limited to, virtual touch or gesture (that is, virtually pressing or selecting virtual buttons or menu items), eye gaze, and / or voice commands. Asfurther discussed below, the physician wearing headset 100 may access a virtual interface 1010a for a particular external device via menu 1010 to select desired settings for a particular external device.

[0042] In a number of embodiments hereof, information / instructions 554 (see FIG. 1A) regarding adjustment (according to settings selected by the physician via virtual interface 1010a) of one or more parameters of an external device 300 transmitted from headset 100 may be transmitted to an external communication device such as device 550, (for example, a tablet computer, a mixed reality headset, etc.; see FIG. 1A) via, for example, a communication system 551 thereof. Such instructions 554 may be communicated to (for example, displayed via a display 552) an individual / operator located out of the sterile field (sometimes referred to herein as an external operator) to carry out instructions 554 via physical manipulation of controls of an external device. After the instructions are carried out via adjustment of the external device, or if such instructions or any portion thereof are not carried out for any reason, the external operator can transmit status information 1100 via tablet 550 to the headset 100 (directly or via communication hub 500) and thereby to the user / physician wearing headset 100 via, for example, holographic or mixed reality display 1000. Such a mode of interaction with an external operator provides a significant improvement over current oral communications, significantly reducing or eliminating problems such as delay, errors, etc.

[0043] As used herein, the term “external communications device” refers to, for example, a portable or mobile device which includes a communication system, a processor system, a user interface system (for example, a visual feedback system including a touchscreen or other display, an auditory feedback system, and a tactile feedback system, a user input system etc.) and an operating system capable of running general-purpose applications. Examples of external communications devices include, but are not limited to, smartphones, tablet computer and custom devices. As used herein, the term “tablet computer” or tablet, refers to a mobile computer with a communication system, a processor system, at least one user interface as described above (typically including a touchscreen display), and an operating system capable of running general-purpose applications in a single unit. As used herein, the term “smartphone” refers to a cellular telephone including a processor system, at least one user interface as described above (typically including a touchscreen display), and an operating system capable of running general-purpose applications. Such communication devices are typically powered by rechargeable batteries and are housed as a single, mobile unit. Moreover, in a number ofembodiments, communications devices are able to accept input directly into a touchscreen (as opposed to requiring a keyboard and / or a mouse) or via voice commands. External communications devices for user herein typically provide for either wired and / or wireless communication as known in the computer arts.

[0044] Alternatively or additionally, a mixed reality headset 550a may be used by an individual located external to the sterile field. Mixed reality headset 550a may, for example, include the same components as described in connection with mixed reality headset 100. Likewise, mixed reality headset 550a may function to provide a holographic or mixed reality display similar to display 1000 and be used as an external communication device in a manner similar to device 550 (for example, a tablet computer) as described above.

[0045] In addition to or in the alternative to information / data regarding adjustment of setting of an external device, an external communication device hereof may receive information / data regarding other types of instructions. For example, device 100 may be configured to provide a menu similar to menu 1010 to provide such information / data to a person located outside the external field regarding or defining such instructions. In a representative embodiment, instructions may be provided regarding surgical equipment to be brought into the sterile field so that the person located external to the sterile field may act as a circulator or assist a circulator as known in the surgical arts.

[0046] In a number of embodiments, system 10 further or alternatively includes an interface system 600 which physically interacts with one or more external devices such as external device 300, 300n to physically adjust or manipulate one or more settings on an external device such as external device 300, 300n in an automated manner based on physician selected setting via virtual interface 1010a. Interface system 600 may, for example, adjust setting of external device 300, 300n and thereby control functionality of one or more aspects of operation of a surgical device such as surgical device 200.

[0047] As illustrated in FIG. 1A, interface system 600 may include a robotic system 700 including electronic circuitry 710. Electronic circuitry 710 may include or be in communicative connection with a communication system 720 to provide for wireless communication (for example, via BLUETOOTH, etc.) and / or wired communication (for example, via USB, ethernet etc.). Electronic circuitry 710 may, for example, be used to control a robotic arm 730 such as an articulated robotic arm 730 as known in the robotic arts. Representative examplesof a commercially available robotic arm systems that can be adapted for use herein include the UFACTORY XARM 6, available from Ufactory Collaborative Robots of Guangdong, P.R. China, and the UFACTORY UR3e, available from Ufactory Collaborative Robots. Robotic arm 730 includes one or more device interface members or systems 740 on a distal end thereof which is / are controllable via electronic circuitry 710 to effect physical manipulation of the various controls of external device 300. An interface system such as interface system 600 and robotic system 700 thereof may, for example, interface with more than one external device 300, 300n. Alternatively, each external device 300, 300n which requires physical manipulation of the controls thereof may have a unique interface system 600, 600n associated therewith.

[0048] In a number of representative embodiments, before system 10 is used intraoperatively (that is, during the course of a surgical operation), communication hub 500 and one or more interface systems 600, 600n may be configured to communicate and perform registration. One or more external devices which may be remotely controlled via direct electronic communication and an external communication device such as external communication device 550, 550a etc. may also be configured to communicate and perform registration. Configuration can be achieved securely using, for example, cryptographic methods (such as digital signatures and secure execution environments). As part of registration, the interface systems 600, 600n may, for example, provide communication hub 500 with information about available data and functionality thereof (including how the stated functionality should be displayed to the user).

[0049] Once interface systems 600, 600n are connected to and registered with communication hub 500, mixed reality headset 100 can also be securely connected to communication hub 500. Upon connecting headset 100 to communication hub 500, information about external device such as external devices 300, 300n associated with interface systems 600, 600n, including their functionalities and a user interface, may be communicated to headset 100. Using such information, headset 100 may construct virtual user interface 1010a that can be displayed to the user intraoperatively via, for example, a holographic or mixed reality interface / display 1000 as described above.

[0050] FIGS. 1C and ID illustrates a representative embodiment of a holographic or mixed reality virtual interface 1010a for use in controlling an external device such as external device 300, 300n or in providing instruct! ons / informati on 554 to an external operator outside the sterile field via communication device 550. In the representative embodiment of FIGS. 1 Aand IB, virtual interface 1010a is associated with an external device for control of electrocautery device 200. Three types of external device control modalities are illustrated in holographic or mixed reality virtual interface 1010a of FIGS. 1C and ID. Referring to the orientation of the figure on the page, on the left of virtual interface 1010a of FIG. 1C, a panel 1012 of preset values is provided. In the middle, a panel 1016 of individual numbers is provided to support direct entry of various selected values. On the right, panel 1020 provides a display box 1022 setting forth the selected value. An up toggle button 1024 and a down toggle button 1026 are provided in the vicinity of display box 1022 for making fine-grain adjustments to the selected value. In a number of embodiments, the selected value in display box 1022 blinks red while the adjustment is being performed and turns steady green when the adjustment is confirmed. If there is an error, a visualization (not shown) of the error may be provided. A selection mechanism 1030 for selecting cutting (CUT; see FIG. 1C) and coagulation (COAG; see FIG. ID) modes is provided on the far right of holographic or mixed reality interface display 1010. Typically, CUT uses a continuous, low-voltage current to vaporize a small area of tissue. The CUT mode is useful for making clean cuts, but is less effective at coagulating. The COAG mode, which is better for coagulating, typically uses an interrupted, high-voltage current to cause tissue to dry out and stop bleeding. Collapse / Expansion buttons 1014 and 1018 can be used to expand (FIG. 1C) and collapse (FIG. ID) panels 1012 and 1016, independently.

[0051] In creating virtual interface 1010a in a number of embodiments hereof, a user may, for example, create an image of external device 300 and automatic detection of buttons and other interface elements that can be used to create a holographic or mixed reality display interface 1010a (see, for example, FIG. 1A) for MR headset 100 via a machine learning algorithm. For example, external device 300 for electrocautery device 200 may include an external, non-sterile interface for adjusting parameters include the frequency and intensity of electrocautery device 200. The user wearing headset 100 may, for example, look at the external device. Headset 100 may be programmed to detect the interface elements (for example, buttons etc.) in a digital image using computer vision. Headset 100 may automatically create holographic or mixed reality virtual interface 1010a for selection of settings / control instructions by the user of headset 100. Alternatively, one may create such virtual interface 1010a manually and connect holographic or mixed reality interface element (for example, buttons etc.) to the specified functionality using, for example, a graphic editor.

[0052] Upon detecting an interaction, headset 100 may send data and commands over a communication channel (for example, via BLUETOOTH, Wi-Fi, etc.) to the communication hub 500 or directly which correspond, for example, to a desired state (that is, a group of settings or setting values) for a selected external device. That information will then be relayed (i) directly to an external device, (ii) to communication device 550, 550a and / or (iii) to interface system 600, 600n associated with the selected external device 300, 300n. Even in the case of communication directly with an external device or to an automated interface system such as interface system 600, 600n, an external operator may also receive instructions via communication system 550, 550a as a redundancy. One of the external device, the external operator, or interface system 600, 600n, will make the designated adjustment to achieve the state selected by the user. As described herein, data binding in system 10 may, for example, be duplex, such that headset 100 may send data and commands to communication hub 500 (and therethrough to, for example, interface systems 600, 600n) and directly controlled external devices, communication device 500, and interface systems 600, 600n may send data and commands to headset 100 through communication hub 500. Interface system 600, 600n, may for example, use computer vision techniques (for example, using machine learning algorithms) to identify target and select the target on external devices 300, 300n for adjustment of settings.

[0053] In a number of embodiments, while operating surgical device 200, the user wearing headset 100 is provided the opportunity to create “safe checkpoints” that act as defaults for the state configuration of external device 300 in the representative cases that device 200 or headset 100 loses connectivity with communication hub 500 or external communication hub loses connection with a directly controlled external device, communication device 550, 550a and / or interface system(s) 600, 600n. This procedure acts as a fail-safe in the case of technical failure. System 10 may also occasionally measure latency in the connection between hub 500 and both headset 100 and a directly controlled external device, communication device 550, 550a and / or interface systems 600, 600n to ensure that the connection is stable. The user may be alerted to the latency while using surgical device 200 via holographic or mixed reality interface / display 1000. If the latency reaches an unsafe threshold, system 10 may, for example, halt active control of external device 300, 300n and the settings thereof (for example, the settings of external device 300 for control of surgical device 200) will be returned to the most recent safe checkpoint state via, for example, the external operator or interface systems 600, 600n.

[0054] FIG. 2 illustrates normal operation for a representative embodiment of system 10. Two interface systems 600, 600n are set forth in the representative embodiment of FIG. 2, which interface with external device 300, and external device 300n as described above. Before initiating system 10, all devices are connected and registered with communication hub 500, which then stores a list of connected devices, their identifiers, and their associated capabilities as described above. At startup, MR headset 100 connects to communication hub 500 and receives information about the devices previously registered with communication hub 500. This may include, but is not limited to, interface design and available control commands for external device 300, 300n. Once information is received about the devices, headset 100 will construct holographic or mixed reality interfaces 1000 for connected external devices 300, 300n and display them to the user.

[0055] FIG. 2 illustrates a representative example of interaction with external device 300 via interface system 600. Although FIG. 2 describes an embodiment including interface system 600, similar interactions and protocols may be established for direct communication with an external device (for example, via loT) and / or communication with an external operator via communication device 550. Once the user interacts with holographic or mixed reality interface / display 1000 (for example, via menu / device interface 1010 and virtual interface 1010a), headset 100 sends the associated command to communication hub 500. Communication hub 500 may subsequently log the command with a timestamp and attempt to relay the command to the appropriate interface system (interface system 600 in this example). Interface system 600 then executes physical manipulation or adjustment of external device 300 associated with the provided command and returns a success or failure message to communication hub 500. Communication hub 500 relays the information to headset 200. The MR headset 100 can then display a success message on the virtual interface 1000, if applicable.

[0056] FIG. 2 also illustrates an embodiment of a safe checkpoint feature. As describe above a safe checkpoint is a device state that is deemed safe to the patient at a certain surgical step. The user can, for example, choose to create a safe checkpoint using the current device state for external device 300 by selecting an option on holographic or mixed reality interface 1000. The create safe checkpoint command is then relayed to communication hub 500, which sends it to the corresponding interface system 600 (for external device 300). Interface system 600 may then cache the current state of settings for external device 300 in a buffer such that, if the connection between interface system 600 and communication hub 500 were to bedisconnected, interface system 600 could return external device 300 to that state to ensure the safety of the patient.

[0057] The bottom section of FIG. 2 (below the line labeled “Connection Lost”) describes an embodiment of system operation for the elements of system 10 in the case communication is interrupted. If communication is interrupted, all external devices 300 may, for example, be returned to their most recently defined safe checkpoint state. Headset 100 may, for example, display a message to the user that the connection has been interrupted, so the user can instruct for manual manipulation the external devices 300, 300n if necessary. Once connection is lost, communication hub 500 may attempt to restore the connection in order to keep interruptions to a minimum.

[0058] In addition to reducing procedure time, increasing safety through, for example, decreasing the risk of human error, the devices, methods and systems hereof may improve the performance of surgical procedure by enabling a surgeon to fine tune the settings of external devices at will without the need to involve another person to effect the changes.

[0059] The foregoing description and accompanying drawings set forth a number of representative embodiments at the present time. Various modifications, additions and alternative designs will, of course, become apparent to those skilled in the art in light of the foregoing teachings without departing from the scope hereof, which is indicated by the following claims rather than by the foregoing description. All changes and variations that fall within the meaning and range of equivalency of the claims are to be embraced within their scope.

Claims

WHAT IS CLAIMED IS:

1. A system for use in connection with a sterile field in a medical setting, comprising:an external device positioned outside of the sterile field;a mixed reality headset comprising a support system for supporting the mixed reality headset upon the head of a user who is located within the sterile field, electronic circuitry comprising a processor system and a memory system in communicative connection with the processor system, the memory system having stored therein one or more software algorithms executable by the processor system to control the mixed reality headset, a display system attached to the support system and in communication with the electronic circuitry, the display system comprising a display, and a communication system in communicative connection with the electronic circuitry, the electronic circuitry being configured (i) to provide a mixed reality display visible to the user via the display system, (ii) to store information regarding adjustment of settings, the settings being for one or more parameters for control of the external device, (iii) to display via the display a virtual interface for the settings through which the user can select values of the one or more parameters to define a state of the settings, and (iv) to transmit instruction data via the communication system of the state of the settings to an external system which is positioned external to the sterile field and via which an adjustment of the one or more parameters may be effected to achieve the state of the settings.

2. The system of claim 1 wherein the external system comprises at least one of (i) a communication system of an external communication device positioned external to the sterile field, the external communication device comprising a user interface to communicate the state of the settings to an external individual, who is located outside of the sterile field, to enable the external individual to physically manipulate controls of the external device to adjust the settings of the external device, (ii) a communication system of the external device, and (iii) a communication system of a system interface, the system interface being positioned outside of the sterile field, the system interface configured to physically manipulate controls of the external device to adjust the settings of the external device.

3. The system of claim 2 wherein the system interface comprises a robotic system.

4. The system of claim 2 wherein the external system is configured to communicate information to the communication system of the mixed reality headset regarding a status of the adjustment of the one or more parameters.

5. The system of claim 2 wherein the external device is in connection with an internal device positioned within the sterile field and at least one of the one or more parameters comprises a control parameter of the internal device.

6. The system of claim 5 wherein the internal device is a powered medical device, and optionally a powered surgical device or a powered diagnostic device.

7. The system of claim 6 wherein the powered medical device is a surgical drill device, a surgical saw device, an electrocautery device, a laser emitting device, a microdebrider device, an endoscope.

8. The system of claim 6 wherein the powered medical device is an electrocautery device.

9. The system of any one of claims 1 through 8 wherein communication between the mixed reality headset and the external system occurs via a communication hub.

10. The system of any one of claims 1 through 8 wherein the electronic circuitry is further configured (i) to store instructional information which is different from the information regarding adjustment of settings, (ii) to display via the display a virtual interface for the instructional information via which the user can select parameters to define one or more instructions, and (iii) to transmit data via the communication system of the one or more instructions to the external system, wherein optionally the one or more instructions comprises information regarding surgical equipment to transport to the sterile field.

11. The system of claim 10 wherein the one or more instructions are transmitted to the communication system of the external communication device.

12. A method for conducting a procedure in a sterile field in a medical setting, comprising:providing a mixed reality headset comprising a support system for supporting the mixed reality headset upon the head of a user located within the sterile field, electronic circuitry comprisinga processor system and a memory system in communicative connection with the processor system, the memory system having stored therein one or more software algorithms executable by the processor system to control the mixed reality headset, a display system attached to the support system and in communication with the electronic circuitry, the display system comprising a display, and a communication system in communicative connection with the electronic circuitry, the electronic circuitry being configured (i) to provide a mixed reality display visible to the user via the display system, (ii) to store information regarding adjustment of settings, the settings being for one or more parameters for control of an external device which is positioned outside the sterile field,displaying via the display a virtual interface associated with the settings via which the user can select values of the one or more parameters to define a state of the settings, andtransmitting instruction data via the communication system of the state of the settings to an external system via which an adjustment of the one or more parameters may be effected to achieve the state of the settings, the external system being positioned external to the sterile field.

13. The method of claim 12 wherein the external system comprises at least one of (i) a communication system of an external communication device positioned external to the sterile field, the external communication device comprising a user interface to communicate the state of the settings to an external individual, the external individual being located outside of the sterile field, to enable the external individual to physically manipulate controls of the external device to adjust the settings of the external device, (ii) a communication system of the external device, and (iii) a communication system of a system interface positioned outside of the sterile field, the system interface configured to physically manipulate controls of the external device to adjust the settings of the external device.

14. The method of claim 13 wherein the system interface comprises a robotic system.

15. The method of claim 12 wherein the external system is configured to communicate information to the communication system of the mixed reality headset regarding a status of the adjustment of the one or more parameters.

16. The method of claim 12 wherein the external device is in connection with an internal device positioned within the sterile field and at least one of the one or more parameters comprises a control parameter of the internal device.

17. The method of claim 16 wherein the internal device is a powered medical device, and optionally a powered surgical device or a powered diagnostic device.

18. The method of claim 17 wherein the powered medical device is a surgical drill device, a surgical saw device, an electrocautery device, a laser emitting device, a microdebrider device, or an endoscope.

19. The method of claim 17 wherein the powered medical device is an electrocautery device.

20. The method of any one of claims 12 through 19 wherein communication between the mixed reality headset and the external system occurs via a communication hub.

21. The method of any one of claims 12 through 19 wherein the electronic circuitry is further configured (i) to store instructional information which is different from the information regarding adjustment of settings, (ii) to display via the display a virtual interface for the instructional information via which the user can select parameters to define one or more instructions, and (iii) to transmit data via the communication system of the one or more instructions to the external system, wherein optionally the one or more instructions comprises information regarding surgical equipment to transport to the sterile field.

22. The method of claim 21 wherein the one or more instructions are transmitted to the communication system of the external communication device.

23. A system for use in connection with a sterile field in a medical setting, comprising:an external device positioned outside of the sterile field;a mixed reality headset comprising a support system for supporting the mixed reality headset upon the head of a user who is located within the sterile field, electronic circuitry comprising a processor system and a memory system in communicative connection with the processor system, the memory system having stored therein one or more software algorithms executableby the processor system to control the mixed reality headset, a display system attached to the support system and in communication with the electronic circuitry, the display system comprising a display, and a communication system in communicative connection with the electronic circuitry, the electronic circuitry being configured (i) to provide a mixed reality display visible to the user via the display system, (ii) to store information regarding instructions to be sent to an external system which is positioned outside of the sterile field, (iii) to display via the display a virtual interface for information regarding the instructions via which the user can select one or more instructions, and (iv) to transmit the one or more instructions via the communication system to the external system.

24. A system for use in connection with an internal field, comprising:an external device positioned outside of the internal field;a mixed reality headset comprising a support system for supporting the mixed reality headset upon the head of a user who is located within the internal field, electronic circuitry comprising a processor system and a memory system in communicative connection with the processor system, the memory system having stored therein one or more software algorithms executable by the processor system to control the mixed reality headset, a display system attached to the support system and in communication with the electronic circuitry, the display system comprising a display, and a communication system in communicative connection with the electronic circuitry, the electronic circuitry being configured (i) to provide a mixed reality display visible to the user via the display system, (ii) to store information regarding adjustment of settings, the settings being for one or more parameters for control of the external device, (iii) to display via the display a virtual interface for the settings through which the user can select values of the one or more parameters to define a state of the settings, and (iv) to transmit instruction data via the communication system of the state of the settings to an external system which is positioned external to the internal field and via which an adjustment of the one or more parameters may be effected to achieve the state of the settings.

25. The system of claim 24 wherein the external system comprises at least one of (i) a communication system of an external communication device positioned external to the internal field, the external communication device comprising a user interface to communicate the state of the settings to an external individual, the external individual being located outside of the internal field, to enable the external individual to physically manipulate controls of theexternal device to adjust the settings of the external device, (ii) a communication system of the external device, and (iii) a communication system of a system interface positioned outside of the internal field, the system interface configured to physically manipulate controls of the external device to adjust the settings of the external device.

26. A mixed reality headset for use in an internal field comprising a support system for supporting the mixed reality headset system upon the head of a user who is located within the internal field, electronic circuitry comprising a processor system and a memory system in communicative connection with the processor system, the memory system having stored therein one or more software algorithms executable by the processor system to control the mixed reality headset, a display system attached to the support system and in communication with the electronic circuitry, the display system comprising a display, and a communication system in communicative connection with the electronic circuitry, the electronic circuitry being configured (i) to provide a mixed reality display visible to the user via the display system, (ii) to store information regarding adjustment of settings, the settings being for one or more parameters for control of an external device positioned outside of the internal field, (iii) to display via the display a virtual interface for the settings through which the user can select values of the one or more parameters to define a state of the settings, and (iv) to transmit instruction data via the communication system of the state of the settings to an external system which is positioned external to the internal field and via which an adjustment of the one or more parameters may be effected to achieve the state of the settings.

27. The mixed reality headset of claim 26 wherein the internal field is a sterile field.

Images