Driving robotic arms utilizing surgical tools to previously saved robotic states

Abstract

A method performed by a surgical robotic system comprising: receiving a save command to save a first robotic state of a robotic arm of a surgical robotic system, the first robotic state including a first orientation and a first position of the robotic arm; saving first state information in response to the save command; driving the robotic arm from the first robotic state to a second robotic state in response to a drive command, the second robotic state being different than the first robotic state and including a second orientation and a second position; receiving a restore command to move the robotic arm from the second robotic state to the first robotic state; and driving the robotic arm to the first robotic state in response to the restore command. Other aspects are also described and claimed.

Description

Docket No. AUR6358WOPCT1 Driving Robotic Arms Utilizing Surgical Tools to Previously Saved Robotic StatesRELATED APPLICATION

[0001] This patent application claims the benefit of the earlier filing date of U.S.Provisional Patent Application 63 / 738,408 filed December 23, 2024, which is incorporated herein by reference in its entirety.FIELD

[0002] Various aspects of the disclosure relate generally to surgical robotic systems, and more specifically to surgical robotic systems that drive robotic arms utilizing surgical tools to previously saved robotic states for surgical procedures. Other aspects are also described.BACKGROUND

[0003] Minimally invasive surgery, MIS, such as laparoscopic surgery, uses techniques that are intended to reduce tissue damage during a surgical procedure. Laparoscopic procedures typically call for creating several small incisions in the patient, e.g., in the abdomen, through which several surgical tools, such as an endoscope, a blade, a grasper, and a needle, are then inserted into the patient. A gas is injected into the abdomen, which insufflates the abdomen, thereby providing more space around the tips of the tools, making it easier for the surgeon to see (via the endoscope) and manipulate tissue at the surgical site. MIS can be performed faster and with less surgeon fatigue using a surgical robotic system in which the surgical tools are operatively attached to the distal ends of robotic arms, and a control system actuates the arm and its attached tool. The tip of the tool will mimic the position and orientation movements of a handheld user input device (UID) as the surgeon is manipulating the latter. The surgical robotic system may have multiple surgical arms, one or more of which has an attached endoscope and others have attached surgical tools or instruments for performing certain surgical actions.

[0004] Control inputs from a user (e.g., surgeon or other operator) are captured via one or more user input devices and then translated into control of the robotic system. For example, in response to user commands, a tool drive having one or more motors may actuate one or more degrees of freedom of a surgical tool when the surgical tool is positioned at the surgical site in the patient. The reach and access of the surgical tool may be limited by the setup of the system.Docket No. AUR6358WOPCT1

[0005] Further, conventional systems can be tedious to operate as surgeons are unable to save important robotic positions and / or orientations. For example, if a surgeon finds positions and orientations of robotic arms for a surgery that they are satisfied with and want to use these again for future surgeries, it may be difficult for the surgeon to quickly recreate that pose. Further, if a fault occurs in the system, it can sometimes be necessary to take down or dismantle the robotic arms to recover from the fault (also known as a system tear down) and start the process over. This may involve moving the robotic arms away from their positions utilized in the surgical procedure to perform the recovery. Once the recovery is completed, the robotic arms can be returned to their previous positions to resume the surgical procedure. However, moving the robotic arms back to their last positions / orientations can sometimes be time-consuming and difficult to perform.SUMMARY

[0006] Implementations of this disclosure include saving known working robotic states of robotic arms (e.g., positions defined by spatial coordinates and orientations defined by joint angles). At the time of saving, the robotic arms may or may not have surgical tools installed to perform a surgical procedure. Thus, in some cases, known working robotic states of robotic arms may be saved while the robotic arms utilize surgical tools to perform a surgical procedure (e.g., a gastrectomy, a gastric bypass, a cholecystectomy, an orthopedic surgery, etc.). Then, a robotic state can be restored based on the state information that was saved. This may enable the robotic state to be used again in a same surgery (e.g., continuing the surgical procedure after recovering from a fault, or repeating the surgical procedure in a next step) or a future surgery (e.g., utilizing the same surgical procedure for the patient or a next patient).

[0007] Some implementations may include a method performed by a surgical robotic system comprising the steps of: receiving a save command to save a first robotic state of a robotic arm of a surgical robotic system, the first robotic state including a first orientation and a first position of the robotic arm and associated surgical tool or instrument, if any present, used to perform a surgical procedure; saving first state information in response to the save command, wherein the first state information includes the first orientation and the first position of the robotic arm; driving the robotic arm from the first robotic state to a second robotic state in response to a drive command, the second robotic state being different than the first robotic state and including a second orientation and a second position; receiving a restore command to move the robotic arm from the second robotic state to the first robotic state; and driving the robotic arm to the first robotic state in response to the restore command,Docket No. AUR6358WOPCT1 wherein the robotic arm is driven to restore the first orientation and the first position based on the first state information.

[0008] Some implementations may include surgical robotic system, comprising: a robotic arm; and a processor configured to: receive a save command to save a first robotic state of the robotic arm of a surgical robotic system, the first robotic state including a first orientation and a first position of the robotic arm and associated surgical tool or instrument, if any present, used to perform a surgical procedure; save first state information in response to the save command, wherein the first state information includes the first orientation and the first position of the robotic arm; drive the robotic arm from the first robotic state to a second robotic state in response to a drive command, the second robotic state being different than the first robotic state and including a second orientation and a second position; receive a restore command to move the robotic arm from the second robotic state to the first robotic state; and driving the robotic arm to the first robotic state in response to the restore command, wherein the robotic arm is driven to restore the first orientation and the first position based on the first state information.

[0009] Some implementations may include method performed by a surgical robotic system comprising the steps of: receiving a save command to save a first robotic state of a robotic arm of a surgical robotic system, the first robotic state corresponding to a starting configuration for a surgical procedure, the first robotic state including a first orientation and a first position of the robotic arm and associated surgical tool or instrument, if any present, used to perform the surgical procedure, wherein the first robotic state results in the surgical tool being a first distance from an anatomical target; saving first state information in response to the save command, wherein the first state information includes the first orientation and the first position of the robotic arm; driving the robotic arm from the first robotic state to a second robotic state in response to a drive command, the second robotic state being different than the first robotic state and including a second orientation and a second position, wherein the second robotic state results in the surgical tool being a second distance from the anatomical target, the second distance being different than the first distance; receiving a restore command to move the robotic arm from the second robotic state to the first robotic state, the restore command corresponding to a stage in which the surgical procedure is repeated; simulating the robotic arm moving along a trajectory to the first robotic state before driving the robotic arm to avoid a collision with an object; and driving the robotic arm to the first robotic state in response to the restore command and after simulating the robotic arm moving along the trajectory, wherein the robotic arm is autonomously guided, upon receivingDocket No. AUR6358WOPCT1 the restore command, from the second position and the second orientation to the first position and first orientation based on the first state information.

[0010] The above summary does not include an exhaustive list of all aspects of the present disclosure. It is contemplated that the disclosure includes all systems and methods that can be practiced from all suitable combinations of the various aspects summarized above, as well as those disclosed in the Detailed Description below and particularly pointed out in the Claims section. Such combinations may have particular advantages not specifically recited in the above summary.BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Several aspects of the disclosure herein are illustrated by way of example and not by way of limitation in the figures of the accompanying drawings in which like references indicate similar elements. It should be noted that references to “an” or “one” aspect in this disclosure are not necessarily to the same aspect, and they mean at least one. Also, in the interest of conciseness and reducing the total number of figures, a given figure may be used to illustrate the features of more than one aspect of the disclosure, and not all elements in the figure may be required for a given aspect.

[0012] FIG. 1 is an exemplary embodiment of a surgical robotic system.

[0013] FIG. 2 an exemplary embodiment of a portion of a robotic arm of the surgical robotic system of FIG. 1.

[0014] FIG. 3A is an exemplary embodiment of a tool drive of the robotic arm of FIG. 2.

[0015] FIG. 3B is an exemplary embodiment of a surgical tool utilized by the robotic arm of FIG. 2.

[0016] FIG. 4 is an exemplary embodiment of structures utilized by the surgical robotic system of FIG. 1.

[0017] FIG. 5 is an exemplary embodiment of a pose of robotic arms for draping with a sterile barrier utilized in the surgical robotic system of FIG. 1.

[0018] FIG. 6 is an exemplary embodiment of a pose of robotic arms to avoid collisions utilized in the surgical robotic system of FIG. 1.

[0019] FIG. 7 is an exemplary embodiment of a pose of robotic arms to increase access to a patient utilized in the surgical robotic system of FIG. 1.

[0020] FIG. 8 is an exemplary embodiment of a pose of robotic arms to perform a surgical procedure utilized in the surgical robotic system of FIG. 1.Docket No. AUR6358WOPCT1

[0021] FIG. 9 is an exemplary embodiment of an offset between a robotic arm and a trocar or patient utilized in the surgical robotic system of FIG. 1.

[0022] FIG. 10 is an exemplary embodiment of a process for driving robotic arms utilizing surgical tools to previously saved robotic states utilizing the surgical robotic system of FIG.1.

[0023] FIG. 11 is an exemplary embodiment of a process for performing a surgical procedure utilizing the surgical robotic system of FIG. 1.DETAILED DESCRIPTION

[0024] Several aspects of the disclosure with reference to the appended drawings are now explained. Whenever the shapes, relative positions and other aspects of the parts described in a given aspect are not explicitly defined, the scope of the disclosure here is not limited only to the parts shown, which are meant merely for the purpose of illustration. Also, while numerous details are set forth, it is understood that some aspects may be practiced without these details. In other instances, well-known circuits, structures, and techniques have not been shown in detail so as not to obscure the understanding of this description. Furthermore, unless the meaning is clearly to the contrary, all ranges set forth herein are deemed to be inclusive of each range’s endpoints.

[0025] Implementations of this disclosure address problems such as these by saving known working robotic states of robotic arms (e.g., positions defined by spatial coordinates and orientations defined by joint angles). At the time of saving, the robotic arms may or may not have surgical tools installed to perform a surgical procedure. Thus, in some cases, known working robotic states of robotic arms may be saved while the robotic arms utilize surgical tools to perform a surgical procedure (e.g., a gastrectomy, a gastric bypass, a cholecystectomy, an orthopedic surgery, etc.). Then, a robotic state can be restored based on the state information that was saved. This may enable the robotic state to be used again in a same surgery (e.g., continuing the surgical procedure after recovering from a fault, or repeating the surgical procedure in a next step) or a future surgery (e.g., utilizing the same surgical procedure for the patient or a next patient).

[0026] Some implementations may include a surgical robotic system including a robotic arm (e.g., one or more robotic arms, such as four robotic arms) and a processor (e.g., one or more processors executing instructions stored in memory). The processor can execute in the system to receive a save command to save a first robotic state of the robotic arm. For example, the save command may be generated via an input device utilized by the operator,Docket No. AUR6358WOPCT1 automatically by the surgical robotic system (e.g., periodically, such as an autosave), and / or in response to detecting a fault. The first robotic state can include a first orientation (e.g., 0 joint angle) and a first position (e.g., X, Y, Z coordinates) of the robotic arm utilizing and associated surgical tool, if any present, to perform a surgical procedure, such as a starting configuration of the arms for a surgical procedure from a first state. The system can store state information of the robotic arm in response to the save command. The state information may indicate a first orientation that includes a first position and a first orientation of the robotic arm, and in some cases, a type of surgical procedure, a type of surgical tool, and / or a profile of the operator. The save command may be received during a stage in which a surgical procedure is set up for an operator of the surgical robotic system.

[0027] The system can then receive a restore command to move the robotic arm from a second robotic state (e.g., a stowed away configuration) back to the first robotic state.Although not limited, the restore command may be received for a stage in which a surgical procedure is repeated. For example, the surgical procedure may be repeated in a same surgery (e.g., continuing the surgical procedure after recovering from a fault, or repeating the surgical procedure in a next step) or a future surgery (e.g., utilizing the same surgical procedure for the patient or a next patient). The system can drive the robotic arm from the second robotic state back to the first robotic state in response to the restore command by the surgical robotic system. The robotic arm may be driven to restore the position and the orientation to utilize the surgical tool based on the state information (e.g., the first robotic state of the robotic arm and surgical tool, and in some cases, the type of surgical procedure, the type of surgical tool, and / or the profile of the operator). In some cases, the robotic arm may be driven to the first robotic state automatically. In some cases, the robotic arm may be driven to the first robotic state based on user input.

[0028] In some cases, the robotic arm may be driven to the first robotic state based on a determined precondition for a safe motion of the robotic arm. For example, a stage in which a precondition for safe motion of the robotic arm is determined may occur after receiving the save command to save the first robotic state, and before driving the robotic arm from the second robotic state back to the first robotic state.

[0029] In some cases, the system can simulate the robotic arm moving along a pre-defined trajectory to the first robotic state before driving the robotic arm to avoid a collision with an object, such as another robotic arm or the table. The system can sense the object along the trajectory to the first robotic state before driving the robotic arm to avoid the collision with the object.Docket No. AUR6358WOPCT1

[0030] Several aspects of the disclosure with reference to the appended drawings are now explained. Whenever the shapes, relative positions, and other aspects of the parts described are not explicitly defined, the scope of the invention is not limited only to the parts shown, which are meant merely for the purpose of illustration. Also, while numerous details are set forth, it is understood that some aspects of the disclosure may be practiced without these details. In other instances, well-known circuits, structures, and techniques have not been shown in detail so as not to obscure the understanding of this description.

[0031] FIG. 1 is an example of a surgical robotic system 1 in an operating environment. The robotic system 1 includes a user console 2, a control tower 3, and one or more surgical robotic arms 4 at a surgical robotic table (surgical table or surgical platform) 5. In one aspect, the arms 4 may be mounted to a table or bed on which the patient rests as shown in the example of FIG. 1. In one aspect, at least some of the arms 4 may be configured differently. For example, at least some of the arms may be mounted on a ceiling, sidewall, or in another suitable structural support, such as a cart separate from the table. The system 1 can incorporate any number of devices, tools, or accessories used to perform surgery on a patient 6. For example, the system 1 may include one or more surgical tools 7 used to perform surgery. A surgical tool 7 may be an end effector that is attached to a distal end of a surgical arm 4, for executing a surgical procedure.

[0032] Each surgical tool 7 may be manipulated manually, robotically, or both, during the surgery. For example, the surgical tool 7 may be a tool used to enter, view, or manipulate an internal anatomy of the patient 6. In an aspect, the surgical tool 7 is a grasper that can grasp tissue of the patient. The surgical tool 7 may be controlled manually, by a bedside operator 8; or it may be controlled robotically, via actuated movement of the surgical robotic arm 4 to which it is attached.

[0033] A remote operator 9, such as a surgeon or other operator, may use the user console 2 to remotely manipulate the arms 4 and / or the attached surgical tools 7, e.g., teleoperation. The user console 2 may be in the same operating room as the rest of the system 1, as shown in FIG. 1. In other environments however, the user console 2 may be in an adjacent or nearby room, or it may be at a remote location, e.g., in a different building, city, or country. The user console 2 may include one or more components, such as a seat 10, one or more foot-operated controls 13 (or foot pedals), one or more (handheld) user-input devices (UIDs) 14, and at least one display 15. The display is configured to display, for example, a view of the surgical site inside the patient 6. The display may be configured to display image data (e.g., still images and / or video). In one aspect, the display may be any type of display, such as aDocket No. AUR6358WOPCT1 liquid crystal display (LCD), a light-emitting diode (LED) display, an organic LED (OLED) display, etc. In some aspects, the display may be a 3D immersive display that is for displaying 3D (surgical) presentations. For instance, during a surgical procedure one or more endoscopic cameras may be capturing image data of a surgical site, which the display presents to the user in 3D. In one aspect, the 3D display may be an autostereoscopic display that provides 3D perception to the user without the need for special glasses. As another example, the 3D display may be a stereoscopic display that provides 3D perception with the use of glasses (e.g., via active shutter or polarized).

[0034] In another aspect, the display 15 may be configured to display at least one graphical user interface (GUI) that may provide informative and / or interactive content, to thereby assist a user in performing a surgical procedure with one or more robotic instruments in the surgical robotic system 1. For example, some of the content displayed may include image data captured by one or more endoscopic cameras, as described herein. In another aspect, the GUI may include selectable UI items, which when manipulated by the user may cause the system to perform one or more operations. For instance, the GUI may include a UI item as interactive content to switch control between robotic arms. In one aspect, to interact with the GUI, the system may include input devices, such as a keyboard, a mouse, etc. In another aspect, the user may interact with the GUI using the UID 14. For instance, the user may manipulate the UID to navigate through the GUI (e.g., with a cursor), and to select an item, the user may hover the cursor over a UI item and manipulate the UID (e.g., selecting a control or button). In some aspects, the display may be a touch-sensitive display screen. In this case, the user may perform a selection by navigating and selecting through touching the display. In some aspects, any method may be used to navigate and / or select a UI item.

[0035] As shown, the remote operator 9 is sitting in the seat 10 and viewing the user display 15 while manipulating a foot-operated control 13 and a handheld UID 14 in order to remotely control the arms 4 and the surgical tools 7 (that are mounted on the distal ends of the arms 4.)

[0036] In some variations, the bedside operator 8 may also operate the system 1 in an “over the bed” mode, in which the beside operator 8 (user) is now at a side of the patient 6 and is simultaneously manipulating a robotically driven tool (end effector as attached to the arm 4), e.g., with a handheld UID 14 held in one hand, and a manual laparoscopic tool. For example, the bedside operator’s left hand may be manipulating the handheld UID to control a robotic component, while the bedside operator’s right hand may be manipulating a manualDocket No. AUR6358WOPCT1 laparoscopic tool. Thus, in these variations, the bedside operator 8 may perform both robotic assisted minimally invasive surgery and manual laparoscopic surgery on the patient 6.

[0037] During an example procedure (surgery), the patient 6 is prepped and draped in a sterile fashion to achieve anesthesia. Initial access to the surgical site may be performed manually while the arms of the robotic system 1 are in a stowed configuration or withdrawn configuration (to facilitate access to the surgical site.) Once access is completed, initial positioning or preparation of the robotic system 1 including its arms 4 may be performed. More about these steps is described herein. Next, the surgery proceeds with the remote operator 9 at the user console 2 utilizing the foot-operated controls 13 and the UIDs 14 to manipulate the various end effectors and perhaps an imaging system, to perform the surgery. Manual assistance may also be provided at the procedure bed or table, by sterile-gowned bedside personnel, e.g., the bedside operator 8 who may perform tasks such as retracting tissues, performing manual repositioning, and tool exchange upon one or more of the robotic arms 4. Non-sterile personnel may also be present to assist the remote operator 9 at the user console 2. When the procedure or surgery is completed, the system 1 and the user console 2 may be configured or set in a state to facilitate post-operative procedures such as cleaning or sterilization and healthcare record entry or printout via the user console 2.

[0038] In one aspect, the remote operator 9 holds and moves the UID 14 to provide an input command to drive (move) one or more robotic arm actuators 17 (or driving mechanism) in the robotic system 1 for teleoperation. The UID 14 may be communicatively coupled to the rest of the robotic system 1, e.g., via a console computer system 16 (or host). The UID 14 can generate spatial state signals corresponding to movement of the UID 14, e.g., position and orientation of the handheld housing of the UID, and the spatial state signals may be input signals to control motions of the robotic arm actuators 17. The robotic system 1 may use control signals derived from the spatial state signals, to control proportional motion of the actuators 17. In one aspect, a console processor of the console computer system 16 receives the spatial state signals and generates the corresponding control signals. Based on these control signals, which control how the actuators 17 are energized to drive a segment or link of the arm 4, the movement of a corresponding surgical tool that is attached to the arm may mimic the movement of the UID 14. Similarly, interaction between the remote operator 9 and the UID 14 can generate, for example, a grip control signal that causes a jaw of a grasper of the surgical tool 7 to close and grip the tissue of patient 6.

[0039] The surgical robotic system 1 may include several UIDs 14, where respective control signals are generated for each UID that control the actuators and the surgical tool (endDocket No. AUR6358WOPCT1 effector) of a respective arm 4. For example, the remote operator 9 may move a first UID 14 to control the motion of an actuator 17 that is in a left robotic arm, where the actuator responds by moving linkages, gears, etc., in that arm 4. Similarly, movement of a second UID 14 by the remote operator 9 controls the motion of another actuator 17, which in turn drives other linkages, gears, etc., of the robotic system 1. The robotic system 1 may include a right arm 4 that is secured to the bed or table to the right side of the patient, and a left arm 4 that is at the left side of the patient. An actuator 17 may include one or more motors that are controlled so that they drive the rotation of a joint of the arm 4, to for example change, relative to the patient, an orientation of an endoscope or a grasper of the surgical tool 7 that is attached to that arm. Motion of several actuators 17 in the same arm 4 can be controlled by the spatial state signals generated from a particular UID 14. The UIDs 14 can also control motion of respective surgical tool graspers. For example, each UID 14 can generate a respective grip signal to control motion of an actuator, e.g., a linear actuator that opens or closes jaws of the grasper at a distal end of surgical tool 7 to grip tissue within patient 6.

[0040] In some aspects, the communication between the surgical robotic table 5 and the user console 2 may be through a control tower 3, which may translate user commands that are received from the user console 2 (and more particularly from the console computer system 16) into robotic control commands that transmitted to the arms 4 on the surgical table 5. The control tower 3 may also transmit status and feedback from the surgical table 5 back to the user console 2. The communication connections between the surgical table 5, the user console 2, and the control tower 3 may be via wired (e.g., optical fiber) and / or wireless links, using any suitable one of a variety of wireless data communication protocols, such as BUUETOOTH protocol. Any wired connections may be optionally built into the floor and / or walls or ceiling of the operating room. The robotic system 1 may provide video output to one or more displays, including displays within the operating room as well as remote displays that are accessible via the Internet or other networks. The video output or feed may also be encrypted to ensure privacy and all or portions of the video output may be saved to a server or electronic healthcare record system.

[0041] To create a port for enabling introduction of a surgical instrument into a patient 6, a trocar assembly (or trocar) may be at least partially inserted into the patient through an incision or entry point in the patient (e.g., in the abdominal wall). In one aspect, the trocar assembly may include a cannula or trocar (e.g., trocar 77A, as shown in FIG. 5), an obturator, and / or a seal. In some variations, the trocar assembly can include an obturator such as a needle with a sharpened tip for penetrating through a patient’s skin. The obturator may beDocket No. AUR6358WOPCT1 disposed within the lumen of the trocar when being inserted into the patient 6. and then removed from the trocar such that a surgical instrument may be inserted through the lumen of the trocar. Once positioned within the body of the patient, the trocar may provide a channel for accessing a body cavity or other site within the patient, for example, such that one or more surgical tools or instruments can be inserted into a body cavity of the patient, as described further herein. It will be understood that a trocar, such as trocar 77A, as described herein includes at least a cannula, and can optionally include an obturator or other components.

[0042] Turning to FIG. 2, a portion of a robotic arm 4 is illustrated according to one aspect of the disclosure. The robotic arm and associated components described herein can form a surgical robotic system according to an aspect of the disclosure. The robotic arm can be incorporated into the surgical robotic system 1, as described herein, or can form a portion of a different system. While a single robotic arm 4 has been illustrated, it will be understood that the robotic arm can include additional arm portions or can be a component of multi-arm apparatus without departing from the disclosure.

[0043] The robotic arm 4 includes a plurality of links (e.g., links 20A - 20E) and a plurality ofjoint modules (e.g., joints 21A - 2 IE), where the links are coupled together by the joint modules (or joints), for example link 20B is coupled to link 20C by joint 21C. In one aspect, the joint modules are for actuating the plurality of links relative to one another. The joints can include various joint types, such as a pitch joint or a roll joint, any of which can be actuated manually or by the robotic arm actuators 17, and any of which may substantially constrain the movement of the adjacent links around certain axes relative to others. The robotic arm 4 also includes a tool drive 23 is attached to a distal end of the robotic arm. As described herein, the tool drive 23 can be configured with a docking (or mating) interface 27 to receive an attachment portion (e.g., a mating interface) of a trocar (e.g., such as attachment portion 94 of trocar 77A, as shown in FIG. 9) such that one or more surgical tools or instruments (e.g., scissors, grasping jaws, endoscopes, staplers, cameras, etc.) can be autonomously guided through a lumen of the cannula of the trocar.

[0044] The plurality of the joint 21A - 2 IE of the robotic arm can be actuated to position and orient the tool drive 23 for robotic surgeries. In one aspect, the positions (e.g., along X-, Y-, and Z-axes) and orientations (e.g., rotational orientations, such as 6 joint angles about the X-, Y-, and Z-axes, e.g., roll, pitch, and yaw) of at least some of the joints and links (e.g., with respect to a reference point, such as the surgical table 5) may define a robotic state of the robotic arm. In some aspects, a robotic arm’s state may include a set ofjoint values of one orDocket No. AUR6358WOPCT1 more of the joints that determines an exact configuration of the robotic arm. In some aspects, as the robotic arm moves (e.g., based on translational and / or rotational movement along at least one of the X-, Y-, and Z- axes), the robotic arm may change between one of several robotic states. In one aspect, the robotic state of a robotic arm indicates (or defines) a position and orientation of a portion of the robotic arm. For instance, the position and orientation may indicate spatial coordinates and angles of a docking interface 27 of a tool drive 23 that is coupled to a distal end of the robotic arm.

[0045] In one aspect, robotic arms of a surgical robotic system (including docking interfaces of tool drives that are coupled thereto) are capable of being positioned and orientated in several different poses. A pose may represent a robotic state of a robotic arm including an orientation and a position of the robotic arm. At least some poses are illustrated in FIGS. 5-8. For example, each of these figures illustrates four robotic arms 74A - 74D, with four tool drives 73A - 73D coupled to distal ends of the robotic arms, respectively, while proximal ends of the robotic arms may be coupled to an object, such as the surgical table 5, a (e.g., movable) cart, etc., each like the robotic arm 4 with tool drive 23 coupled to a distal end thereto. Surgical tools like the surgical tool 28 may be coupled to the drives 73A - 73D. Each of these figures also include a patient 6 disposed (or positioned) on top of a tabletop 75 of the surgical table 5. Inserted into the abdomen of the patient are four trocars 77A - 77D.

[0046] FIG. 5 illustrates each of the robotic arms in a “drape” pose or configuration, which is a configuration of the arms 4 in which an operator 78 may drape at least a portion of each robotic arm with a sterile barrier (not shown) to minimize, inhibit, or prevent the transmission of pathogens. FIG. 6 illustrates each of the robotic arms in a “candle” pose or configuration, which is a configuration of the arms 4 in which a distal end of the robotic arm is folded and raised (e.g., with respect to the position of the arms in the drape pose). In one aspect, this pose may be utilized for one or more arms to avoid collisions between the robotic arms and other objects during execution of a planned trajectory, for example to restore a position and an orientation as described herein (e.g., upon receiving the restore command, automatically repositioning the robotic arm to the robotic state such that the robotic arm is autonomously guided from to first position and orientation). This figure also illustrates a (first) open area 81 that extends from (or is between) arm 74A and arm 74C to provide room to the operator 78 to access the patient.

[0047] FIG. 7 illustrates each of the robotic arms in a “preparation” pose or configuration, which is a configuration of the arms 4 that increases or maximizes physical and direct access to the patient, while maintaining sterility of the draped robotic arms (e.g., draped while in theDocket No. AUR6358WOPCT1 drape pose). For example, this figure shows a (second) open area 91 that is between arm 74A and arm 74C to provide increased access to the patient that is larger than area 81 illustrated in FIG. 6.

[0048] FIG. 8 illustrates each of the robotic arms in a surgical “procedure” pose or configuration, which is a configuration of the arms 4 in which the operator 78 may manually guide (e.g., while being actively assisted by the robotic arm actuators 17 under control of a processor, e.g., to augment or amplify manual forces applied by the operator) the robotic arm to couple to a trocar that is inserted into the patient. In one aspect, the procedure pose is a procedure-specific and / or operator-specific pose, such as to maximize reach and access of the surgical workspace area for the operator while minimizing the likelihood of inter-robot collisions during teleoperation. Thus, there may be many different procedure poses for the many different possible surgical procedures and / or operators that may utilize the system 1. In one aspect, the procedure pose is a predefined pose that is surgical procedure and / or operator specific that is independent of anthropometric traits of patients that are to be positioned on the surgical table. In other words, the pose may be the same, regardless of the size and shape of a patient who is positioned on the table.

[0049] In one aspect, when a robotic arm is in a (predefined) procedure pose of FIG. 8, the arm (or at least a portion of the arm) may be at (or within) an offset or threshold distance from an anatomical target or trocar that is coupled to a patient who is laying on the tabletop 75. Specifically, while each of the robotic arms is in its respective procedure pose, each of the arm’s docking interface is equal to or less than a threshold distance from an anatomical target trocar. For example, FIG. 9 is a magnified view A-A of FIG. 8, showing the docking interface 27 of tool drive 73A coupled to robotic arm 74A, and surgical tool 28A coupled to the tool drive 73A, that is at a threshold distance (D) from trocar 77A or anatomical target. Specifically, the docking interface 27 is within D of an attachment portion 94 of the trocar, which protrudes from the trocar and is configured to (removably) couple to the docking interface 27. In some aspects, each of the procedure poses of the robotic arms 74A - 74D may bring each of their respective tool drives 73A - 73D a (same or different) distance (such as D) from a respective trocar or anatomical target. In other words, each robotic arm may have a corresponding procedure pose in which a respective docking interface is moved within a respective threshold distance of a respective trocar or anatomical target.

[0050] In another aspect, along with (or in lieu of) the docking interface being within the threshold distance, each robotic arm’s procedure pose of FIG. 8 may maximize distances between other robotic arms (and / or other objects within the surgical room). For example, theDocket No. AUR6358WOPCT1 links 20A - 20E and / or the joints 21A - 21E of robotic arm 74A may be separated from (e.g., corresponding, or different) links and joints of each other robotic arm 73B - 73D as much as possible. This may ensure that each of the robotic arms has sufficient space between other robotic arms to prevent arms from bunching up.

[0051] In another aspect, the procedure pose of FIG. 8 may include a first orientation and a first position of the robotic arm. For example, the procedure pose of robotic arm 74A is a pose of (e.g., a docking interface of) tool drive 73A (position and / or orientation) that at least partially matches a pose of a corresponding trocar, which in this case would be trocar 77A. In some aspects, the pose of the tool drive 73A and the trocar 77A may at least partially match, while the tool drive is positioned at the distance, D, described above. As described herein, once in this pose the operator may manually guide the robotic arm the rest of the way to latch the tool drive to the trocar.

[0052] In one aspect, the surgical robotic system may include other poses not illustrated herein. For example, at least one of the robotic arms may be in a stowed away (“stow”) pose or configuration, in which the robotic arm is in a folded configuration and stowed under the tabletop 75 of the surgical table 5. While each of the arms are in this pose, patients may access (e.g., lay on) the surgical table without bumping into or colliding with the arms. In one aspect, each of the robotic arms may be positioned in the stow pose before and / or after each surgical procedure.

[0053] In one aspect, the robotic arms may transition between one or more of these poses, including between different procedural poses, while the surgical robotic system drives (e.g., one or more actuators of) the arms along a planned trajectory. More about each of the poses illustrated in FIGS. 5-8 and the robotic arms saved in positions corresponding to poses is described herein.

[0054] FIG. 3A is a schematic diagram illustrating an exemplary tool drive 23 without a loaded surgical tool in accordance with aspects of the subject technology. In one aspect, the tool drive 23 may include an elongated base (or “stage”) 24 having longitudinal tracks 25 and a tool carriage 26, which is slidingly engaged with the longitudinal tracks 25. The stage 24 may be configured to couple to a distal end of a robotic arm 4 such that articulation of the robotic arm 4 positions and / or orients the tool drive 23 in space. The tool carriage 26 may be configured to receive a surgical tool for extending through a trocar.

[0055] Additionally, the tool carriage 26 may actuate a set of articulated movements through a cable system or wires manipulated and controlled by actuated drives (the terms “cable” and “wire” are used interchangeably throughout this application). The tool carriageDocket No. AUR6358WOPCT1 26 may include different configurations of actuated drives, such as a mechanical transmission.

[0056] FIG. 3B is a schematic diagram illustrating an exemplary surgical tool 28 that may be loaded with the tool drive 23 in accordance with aspects of the subject technology. The surgical tool 28 may be received by the tool carriage 26 for extending through the trocar. For example, the surgical tool 28 may include an end effector 30 at its distal end for extending through the trocar to perform a surgical procedure with the anatomy of a patient. In various implementations, the surgical tool 28 could implement scissors, grasping jaws (shown in FIG.3B by way of example), an endoscope, a stapler, a camera, or another instrument that may be utilized to perform the surgical procedure.

[0057] FIG. 4 is a block diagram of the surgical robotic system 1 that drives robotic arms 4 and surgical tools to previously saved robotic states for surgical procedures according to one aspect. The system 1 may include a controller 40, one or more input devices 41, a state database 43 (e.g., a storage, file, or memory), one or more sensors 44, and / or one or more robotic arms 4. The controller 40 may include a state controller 45, a trajectory planner 46, and / or a control signal generator 47. In various aspects, the system 1 may include more or less components. For example, the system 1 may include two or more input devices 41, and / or might not include sensors 44.

[0058] As described herein, a processor (e.g., one or more processors executing instructions stored in memory, such as a processor of the controller 40) can save known, working states of robotic arms 4 (e.g., a position and an orientation, such as the configuration of FIG. 8) to the state database 43. The processor can save the states while the robotic arms 4 are utilized to perform a surgical procedure (e.g., a gastrectomy, a gastric bypass, a cholecystectomy, an orthopedic surgery, etc.). The processor can then enable the robotic arms 4 to restore the position and the orientation to utilize the surgical tool based on the state information that was saved (e.g., upon receiving a restore command, automatically repositioning the robotic arm to a state such that the robotic arm is autonomously guided from to the position and the orientation). This may enable the robotic state to be used again in a same surgery (e.g., continuing the surgical procedure after recovering from a fault, or repeating the surgical procedure in a next step) or a future surgery (e.g., utilizing the same surgical procedure for the patient or a next patient).

[0059] In one aspect, the sensor 44 may be any type of sensor that is arranged to sense (or detect) environmental characteristics in or around the operating room (e.g., as illustrated in FIG. 1). In some implementations, the sensor 44 may include one or more markers on theDocket No. AUR6358WOPCT1 body of a patient, utilized by an optical motion capture system of the surgical robotic system 1. The sensor 44 may be configured to generate sensor data (e.g., as electrical signals) to components of the controller 40 to indicate the characteristics. For example, the sensor may be an optical sensor, such as a camera that is arranged to capture image data of a scene within a field of view of the camera as one or more images. Specifically, the camera’s field of view may include at least a portion of the surgical table 5, one or more of the robotic arms 4, and / or an area surrounding the table and arms. In one aspect, the camera may be a three-dimensional (3D) camera that is configured to capture images and render them in 3D, providing realistic depth. As another example, the sensor may include a motion sensor that is arranged to sense motion (e.g., of a user, of one or more robotic arms, etc.), and is configured to produce motion data that represents such motion. In another aspect, the sensor may include a proximity sensor (e.g., an inductive sensor, a capacitive sensor, etc.) that is configured to detect the presence or absence of objects. In another aspect, the sensor may be a microphone that is arranged to capture ambient sounds as a microphone signal.

[0060] In one aspect, the system 1 may include sensors 44 that are separate electronic devices positioned about the operating room, or more specifically, about the surgical table 5, as illustrated in FIG. 1. In another aspect, at least some of the sensors may be a part of one or more components, such as the surgical table and one or more of the robotic arms. For instance, the robotic arms 4 may include one or more sensors, such as a force / torque (F / T) sensor that is configured to receive, as inputs, forces or torques that have been manually exerted, on the robotic arm 4 by an operator (or an object, such as another robotic arm that comes into contact with a robotic arm that includes the F / T sensor), and to produce corresponding electrical signals as outputs (e.g., to the controller 40). The F / T sensor can also receive, as inputs, forces excreted on the robotic arm by the robotic arm actuators 17.Accordingly, the F / T sensor can be configured to receive, as inputs, linear forces, or rotational forces, e.g., torque. In one aspect, one or more F / T sensors may be mounted or integrated in one or more portions (e.g., one or more joints and / or one or more links) that make up the robotic arm 4. In another aspect, the robotic arm may include sensors that indicate the position and / or orientation of at least a portion of the robotic arm. For instance, the arm may include a motion sensor (e.g., an inertial measurement unit (IMU) that is configured to measure (or sense) motion and / or orientation of at least a portion of the robotic arm and generate sensor data that indicates the motion (or position) and orientation of the arm. In another aspect, the arm may include proximity sensors that indicate the position of the arm (e.g., with respect to other objects and / or other portions of the robotic arm).Docket No. AUR6358WOPCT1

[0061] In one aspect, the input device 41 may be any type of electronic device that is configured to communicate with the controller 40 and receive user input from the operator. For example, the input device may be a computer with one or more input peripheral devices, such as a mouse, keyboard, touch-sensitive display screen, etc. In another aspect, the input device may be a portable electronic device, such as a tablet computer, a laptop, a smart phone, or a remote control. In one aspect, the input device 41 may be a remote control, the user console 2, the control tower 3, or UID 14, and may include an input button, switch, or other physical actuator or modality integrated with the system 1. In one aspect, the input device 41 may be configured to establish a communication link with the controller 40 in order to exchange data with components thereof. For example, the link may be a wired link (e.g., via optical fiber) and / or a wireless link, using any wireless protocol. As illustrated in FIGS. 5-8, the input device 41 (which is illustrated as a tablet computer) may be located (and held by the operator 78) in the operating room. In another aspect, the input device may be at another location remote from the operating room (and from the surgical table).

[0062] The state controller 45 may control operation of the state database 43. The state database 43 is a storage, file, or memory that saves robotic states of one or more of the robotic arms 4 (corresponding to positions and orientations), including but not limited to the mappings to types of surgical procedures, surgical tools, and / or profiles of operators. As described herein, a position may indicate spatial coordinates of joints and / or links corresponding to a robotic arm 4 (including a surgical tool) in a robotic state (e.g., Cartesian coordinates on X, Y, Z axes relative to a reference point). Also, an orientation may indicate joint angles of joints corresponding to a robotic arm 4 (including a surgical tool) in a robotic state (e.g., rotational orientations, such as 6 joint angles about the X-, Y-, and Z-axes, e.g., roll, pitch, and yaw). For example, the position and orientation may indicate spatial coordinates and joint angles associated with each of links 20A - 20E and / or joints 21A - 21E of a robotic arm 4 (e.g., illustrated in FIG. 2) and / or surgical tool 28, e.g., end effector 30 (e.g., illustrated in FIG. 3B).

[0063] As illustrated, the state database 43 may include data structures that save, for a given surgical procedure and / or a given operator, one or more robotic states that may be used (by the system 1) to restore a position and an orientation of one or more of the robotic arms 4. The data structures may include saved positions and orientations of the robotic arms 4 (e.g., arms 74A - 74D, as illustrated in FIG. 8). For example, for a first surgical procedure (e.g., a gastrectomy), mapped to a profile of a first operator, a data structure 48A can include a saved robotic state that defines a position and an orientation for each of the four arms (e.g., Posl 1-Docket No. AUR6358WOPCT1 Posl4). The positions and orientations may be different from one another for each robotic arm 4 (or a portion of the arm) to arrange and orientate the arm 4 at a respective location. This may enable later to restore the positions and orientations for the first surgical procedure. For a second surgical procedure (e.g., an orthopedic surgery), also mapped to the profile of the first operator, the data structure 48A can include a saved robotic state that defines a position and an orientation for the first three arms (e.g., Pos21-Pos23), while not having a position or orientation for the fourth arm (represented as an “X”). In this case, the second surgical procedure may only require three of the four arms, and therefore the fourth arm may remain in a different, default position and orientation (e.g., a stowed away configuration, or one of the other configurations illustrated in FIGS. 5-7). Similarly, for a second operator, a data structure 48B, mapped to a profile of the second operator, may include saved robotic states for different procedure configurations of the robotic arms 4, and so forth.

[0064] In operation, the state controller 45 can utilize a processor to receive a save command to save a first robotic state of the one or more robotic arms 4. For example, the save command may be generated via an input device 41 utilized by the operator, automatically by the controller 40 (without receiving human input from an input device), and / or in response to detecting a fault in the system 1. The first robotic state can include a first orientation and a first position of the one or more robotic arms 4 utilized to perform a surgical procedure. For example, this could be a starting configuration for a surgical procedure (e.g., a configuration of the one or more robotic arms 4 when coupled to one or more surgical tools 28, such as scissors, grasping jaws, endoscopes, staplers, or cameras, via tool drives 23 attached to trocars, e.g., the procedure pose illustrated in FIG. 8). The first robotic state may be mapped to the first operator in the data structure 48A. The processor can then save or store state information of the one or more robotic arms 4 to the data structure in response to receiving the save command. The state information may indicate the first orientation and the first position of the one or more robotic arms 4 (e.g., Posl 1-Posl4, corresponding to the first surgical procedure), the type of surgical procedure, the type of surgical tool, and / or the profile of the operator.

[0065] The processor can then move from the saved robotic state to another robotic state in response to receiving a drive command (which can also be saved). The processor can then execute to receive, e.g., at a future surgery or after recovering from a fault, a restore command to move the one or more robotic arms 4 from a second robotic state back to a saved robotic state (e.g., the first orientation and the first position, saved in the state database 43). For example, the restore command may be generated via an input device 41 utilized by theDocket No. AUR6358WOPCT1 operator, automatically by the controller 40 (without receiving human input from an input device), and / or in response to recovering from a fault in the system 1. The restore command may be received for a stage in which a surgical procedure is repeated. The surgical procedure may be repeated in a same surgery (e.g., continuing the surgical procedure after recovering from a fault, or repeating the surgical procedure in a next step) or a future surgery (e.g., utilizing the same surgical procedure for the patient or a next patient). The processor can drive the one or more robotic arms 4 from the second robotic state back to the first robotic state in response to the restore command. The one or more robotic arms 4 may be driven to restore the position and the orientation to utilize the surgical tool 28, including with the end effector 30 at the desired position and orientation, based on the state information in the data structure 48A (e.g., Posl 1-Posl4).

[0066] In some cases, the one or more robotic arms 4 may be driven to the first robotic state based on determining the first robotic state to be a safe orientation and position for the one or more robotic arms 4 (e.g., a predetermined safe robotic state) and the trajectory or path to the first robotic state being a safe trajectory or path. For example, the first robotic state may be determined to be safe before driving the one or more robotic arms 4 from the second robotic state back to the first robotic state. The processor can determine that it is safe for one or more robotic arms 4 to move to the commanded, previously saved robotic state (e.g., Posl 1 for Arm 1) from an initial robotic state (e.g., a stowed configuration or pose). The processor can determine the feasibility of driving the arms 4 to the robotic state saved in the state database 43.

[0067] In some cases, the processor can execute to simulate the one or more robotic arms 4 moving along a trajectory to the first robotic state before driving the one or more robotic arms 4 to avoid a collision with an object, such as another robotic arm 4 or the table. The processor can execute to sense the object along the trajectory to the first robotic state before driving the one or more robotic arms 4 to avoid the collision with the object.

[0068] In some cases, the processor may enable general position storing and loading for operator convenience. For example, during normal operation, with a robotic arm 4 at a given robotic state, the state controller 45 can save the state to the state database 43 (e.g., data structure 48A). This could be performed via input device 41, such as a remote control, the user console 2, the control tower 3, or UID 14, or some other user-controlled means, which may include an input button, switch, or other physical actuator or modality integrated with the system 1. When the operator interacts with the input device 41 and issues the save command, the system can read information about the position and the orientation of eachDocket No. AUR6358WOPCT1 robotic arm 4 (spatial coordinates, joint angles, etc.) and save that information to the state database 43 (e.g., Posl 1 for Arm 1). This can be saved globally, or to a specific profile associated with an operator (e.g., the first operator). In future use or in subsequent boots of the system, this position and orientation can be read from the saved robotic state in the state database 43 to command the robotic arm 4 to move to the position and orientation and restore the corresponding configuration. For example, the restore command can be selected by a procedure or file name, and / or may be commanded by the operator via the input device 41. The processor can drive the robotic arms 4 to the robotic state based on this user interaction.

[0069] In some cases, the processor can drive each arm to a position and orientation that is offset (e.g., above, or outside of a saved position and orientation) relative to the patient or the trocar. The operator can then correct for any case-to-case variation. For example, the processor can drive to a position and orientation corresponding to the threshold distance (D) from a corresponding trocar or anatomical target that is coupled to a patient who is laying on a tabletop, as illustrated in FIG. 9. The operator can then complete the attachment. Further, in some cases, the processor can modify the commanded, a previously saved robotic state based on the environment. For example, the processor can modify the position and orientation to accommodate bedside assistant height, table height, trends, or other positions or objects of the system 1.

[0070] In some cases, the processor can enable robotic state storing in response to an error or fault. For example, in response to detecting a fault, the processor can save the last known robotic state of robotic arms 4 to a data structure of the state database 43. The processor can execute to resolve the fault, including with the operator taking down or dismantling the robotic arms 4 from their current pose (e.g., removal from the patient and / or trocar, detachment of surgical tools, etc., e.g., a tear down). As the system 1 is being restored, an interaction with the input device 41 can indicate whether the user wants to restore the previous robotic state (the previous position and the previous pose) of the robotic arms 4 from before the fault. For example, this could correspond to a prompt, screen indicators, and / or a simulated view of where the robotic arms 4 are presently, and where the robotic arms 4 will attempt to travel. In some cases, this may include the controller 40 performing a simulation to show future positions of the robotic arms 4 to restore the position and the orientation (e.g., Posl 1-Posl4). During a surgical procedure, the processor can load the robotic state from the state database 43 and drive the robotic arms 4 to the robotic state. In some cases, this can be performed automatically after the operator presses a button of the input device 41 to issue the restore command. In some cases, this can be done with a “pressDocket No. AUR6358WOPCT1 and hold” feature corresponding to the restore command, which indicates permission for the controller 40 to move the robotic arms 4 to the robotic state so long as the button of the input device 41 continues to be pressed. In some cases, this can be done without receiving human input from an input device.

[0071] In some aspects, the robotic state that is loaded can include an insertion depth associated with a surgical tool 28 (e.g., end effector 30 of scissors, grasping jaws, endoscopes, staplers, or cameras) within the body of a patient. For example, the state information in the data structure may further indicate the insertion depth of the surgical tool or end effector to restore the position and the orientation to perform the surgical procedure. This can enable the same surgical tools to be reattached to the robotic arms 4 and placed at the same depth within the body of the patient. In some cases, the processor can enable adjustments based on user interaction to place the surgical tools at different insertion depths as desired.

[0072] In some aspects, each robotic arm 4 may include an indicator 50 or other marker coupled to the robotic arm (see, e.g., FIG. 2). In some cases, the indicator 50 may be a visual indicator to provide a visual indication to surgical staff, including, for example, an LED ring that is at least partially circumferentially disposed around the robotic arm (e.g., partially or completely around the arm). The LED ring can change color after the robotic arm 5 is driven to a position corresponding to a stage. For example, the LED ring can change color when the surgical procedure is set up, when the surgical procedure is repeated, and / or when a precondition for safe motion of the robotic arm is determined before driving the robotic arm. In some cases, the visual indication may be output to a user display, such as the user display 15, or a display 51 or GUI on the robotic arm.

[0073] In some aspects, the system can utilize a machine learning model to predict the robotic state (e.g., orientation and position) or pose that the robotic arm 4 should move to, including from among a plurality of saved robotic states in the state database 43. The machine learning model can predict the robotic state based on data, such as sensor data from sensors 44, the specific operator (e.g., according to their profile), and / or states determined to be safe and / or unsafe position. For example, the machine learning model may be trained based on sensor data associated with one or more operators and / or one or more surgical procedures. The machine learning model may, for example, be or include one or more of a neural network (e.g., a convolutional neural network, recurrent neural network, deep neural network, or other neural network), decision tree, vector machine, Bayesian network, cluster-Docket No. AUR6358WOPCT1 based system, genetic algorithm, deep learning system separate from a neural network, or other machine learning model.

[0074] Thus, one or more robotic states and / or preconditions may apply to driving the robotic arms 4 to robotic states that have been saved, whether pursuant to non-faulted conditions (e.g., for convenience in a future procedure) or faulted conditions (e.g., errors involving a recovery process and possible tear downs). This may include the controller 40 determining that it is safe for robotic arms 4 to move to a position and orientation (e.g., the saved robotic state) from another position and orientation (e.g., an initial robotic state, such as the tear down or stowed away configuration). The processor can enforce one or more preconditions for the safe motion, including sensing conditions in the environment and / or performing simulations of arm movements to determine safe travel.

[0075] As a result, the system may reduce the setup time to utilize poses (positions and orientations) of robotic arms 4 that may be preferred by an operator. The system may also enable saving and restoring poses of robotic arms 4 based on preferences of the operator. The system may also enable recovering a robotic arm 4 from a fault by quickly restoring a prefault pose. This can reduce the surgical procedure time by speeding up recovery from the fault.

[0076] The trajectory planner 46 can utilize a processor to determine one or more planned trajectories for one or more of the robotic arms 4 to robotic states, including, to restore a position and an orientation. A planned trajectory may include a path along which (e.g., a portion of) a robotic arm 4 moves and is orientated by the surgical robotic system, or more specifically by one or more actuators 17 of the robotic arm, in order to automatically (e.g., without user intervention or human input) drive the arm to the desired position and orientation (e.g., a saved robotic state). For example, a planned trajectory may define a path along which a (docking interface 27 of a) tool drive 23 coupled to a distal end of a robotic arm 4 moves from one position and orientation in space to another.

[0077] In some aspects, the planned trajectories may be predefined paths in which one or more of the robotic arms 4 are driven along respective planned trajectories (either consecutively or simultaneously), so that no portion of the robotic arm 4 comes into contact with 1) at least 95% of patients who may be positioned on the tabletop 75 of the surgical table 5 for a surgical procedure associated with the planned trajectory, 2) any other portion of other robotic arms of the surgical robotic system, 3) and other objects that are within a range of motion of the robotic arm, such that the robotic arm does not collide with other arms, theDocket No. AUR6358WOPCT1 patient, or objects while being driven along the planned trajectory. In one aspect, the planned trajectories may be predefined to avoid collisions.

[0078] The control signal generator 47 can utilize a processor to signal one or more robotic arms 4 to drive (or control) the arms along their respective planned trajectories to positions, including to restore a robotic state. In particular, the generator 47 is configured to receive planned trajectories from the trajectory planner 46 and generate control signals to drive the robotic arms 4 along their respective planned trajectories from their respective current robotic state to a robotic state for a respective predefined procedure pose (e.g., a saved robotic state). In one aspect, the controller 40 may simultaneously (or consecutively) drive each of the robotic arms along their respective planned trajectories to respective predefined procedure poses. In one aspect, the controller 40 can use the control signals generated by the generator 47 to signal one or more actuators 17 of one or more robotic arms 4 to drive a segment, joint, or link of the arm to move and / or orientate according to an arm’s respective planned trajectory. Once a robotic arm 4 is positioned and orientated into its respective procedure pose, the controller 40 may pause the arm in place.

[0079] In one aspect, the generator 47 is configured to receive sensor data from one or more sensors 44 and drive the robotic arms based on the sensor data. For example, while driving (e.g., signaling at least one actuator of) a robotic arm to traverse a planned trajectory to a position, the generator may pause the robotic arm if sensor data indicates that an object is going to (or has) collided with the arm. As described herein, one or more sensors (e.g., F / T sensors) may be integrated within the robotic arms. While a robotic arm is being driven along its planned trajectory, F / T sensors (or contact sensors) may sense and generate data (signals) upon the arm coming into contact with an object. Upon determining that the arm is in contact with an object, the generator may pause the arm and / or reverse the arm along the planned trajectory to avoid any further contact. In another aspect, upon determining that an arm is going to come into contact with an object, the trajectory adjustor may adjust a planned trajectory (e.g., in real time), in which case the generator may adjust the movement of the robotic arm in order to avoid collisions. In another aspect, the sensor data may include one or more images captured by one or more cameras of the surgical robotic system. Upon determining that an arm is going to come into contact with an object within the image, the generator may pause the arm.

[0080] In some aspects, the generator 47 may be configured to signal the robotic arms 4 to traverse along their respective planned trajectories to their positions based on user input at the input device 41. For instance, the input device includes an input 79, which is illustrated inDocket No. AUR6358WOPCT1 FIG. 5 as a physical input, such as a button or switch. In one aspect, the input may be a GUI item displayed on a (e.g., touch-sensitive) display screen of the input device. When the input is selected (and held) by an operator, the input device is configured to generate a control signal that controls the operation of the generator 47. In particular, while the input device generates the control signal in response to the input being held by the operator, the generator 47 may (continue) to produce control signals and the controller 40 may continue driving the robotic arms 4 according to the control signals along their respective planned trajectories. Thus, so long as the input device receives user input, the surgical robotic system drives the robotic arms. In response to the input device not receiving user input, however, (e.g., upon a user releasing the input 79) the generator may pause each of the robotic arms at an intermediate pose between an initial pose at which the arm begins to traverse the planned trajectory and a respective predefined procedure pose (e.g., a saved robotic state). In one aspect, upon receiving user input again at the input device, the generator may continue signaling the robotic arms to traverse their plarmed trajectories.

[0081] Reference is now made to flowcharts of examples of processes for driving robotic arms utilizing surgical tools to previously saved robotic states. The processes can be executed using computing devices, such as the systems, hardware, and software described with respect to FIGS. 1-9. The processes can be performed, for example, by executing a machine-readable program or other computer-executable instructions, such as routines, instructions, programs, or other code. The operations of the processes or other techniques, methods, or algorithms described in connection with the implementations disclosed herein can be implemented directly in hardware, firmware, software executed by hardware, circuitry, or a combination thereof.

[0082] For simplicity of explanation, the processes are depicted and described herein as a series of operations. However, the operations in accordance with this disclosure can occur in various orders and / or concurrently. Additionally, other operations not presented and described herein may be used. Furthermore, not all illustrated operations may be required to implement a process in accordance with the disclosed subject matter.

[0083] FIG. 10 is an example of a process 100 for driving robotic arms utilizing surgical tools to previously saved robotic states utilizing the surgical robotic system of FIG. 1. At operation 102, a processor (e.g., one or more processors executing instructions stored in memory, such as a processor of the controller 40) in the surgical robotic system 1 can operate a robotic arm 4 (e.g., one or more robotic arms, such as four robotic arms) to perform a surgical procedure. For example, the robotic arm 4 may be controlled by an operator viaDocket No. AUR6358WOPCT1 teleoperation, such as to perform a gastrectomy, a gastric bypass, a cholecystectomy, an orthopedic surgery, etc. The robotic arm 4 may be configured in a robotic state to perform a surgical procedure with a surgical tool 28, such as a starting configuration for a surgical procedure. For example, the robotic state may include a position and orientation to enable the operator to reach certain anatomy in the body of a patient, via the surgical tool coupled to a tool drive attached to a trocar (e.g., FIG. 9).

[0084] The processor can receive a save command to save a first robotic state of the robotic arm 4. The first robotic state can include a first orientation and a first position of the robotic arm 4, which may be utilizing a surgical tool 28 to perform the surgical procedure (e.g., the starting configuration). For example, the save command may be generated via an input device 41 utilized by the operator, automatically by the surgical robotic system (e.g., periodically, such as an autosave, without receiving human input from an input device), and / or in response to detecting a fault. The first position may correspond to a configuration of one or more robotic arms 4, including when coupled to surgical tools, such as scissors, grasping jaws, endoscopes, staplers, cameras, etc., via tool drives attached to trocars.

[0085] The processor can execute to save or store state information of the robotic arm 4 in response to the save command. The state information may be saved to a data structure in the state database 43 (e.g., data structure 48A). The state information may indicate the first robotic state of the robotic arm 4, and in some cases, the type of surgical procedure, the type of surgical tool, and / or the profile of the operator. For example, the state information may indicate joint angles and / or spatial coordinates associated with each of a plurality of links (e.g., links 20A - 20E) and / or a plurality of joint modules (e.g., joints 21A - 21E) of the robotic arm 4. In some cases, the state information may indicate an insertion depth of the surgical tool (e.g., end effector, within the body of a patient) that may be utilized by the robotic arm 4 to perform the surgical procedure. In some cases, the state information may correspond to a profile of the operator of the surgical robotic system 1 (e.g., a first operator, a second operator, etc.). In some cases, the state information may correspond to a type of surgical procedure to be performed (e.g., a gastrectomy, a gastric bypass, a cholecystectomy, an orthopedic surgery, etc.). In some cases, the state information may correspond to a type of surgical tool used (e.g., scissors, grasping jaws, endoscopes, staplers, cameras, etc.). The save command may be received during stage in which the surgical procedure is set up for the operator of the surgical robotic system 1.

[0086] In some cases, at operation 104 the processor may detect an error or fault in the system 1 while performing the surgical procedure. In response to detecting a fault, theDocket No. AUR6358WOPCT1 processor may execute to save the state information corresponding to a last known position of the robotic arm 4 to the state database 43 (e.g., data structure 48A). For example, the processor may automatically execute to generate the save command in response to detecting the fault (without receiving human input from an input device), and / or output a prompt to the operator (e.g., via display 15 or other feedback mechanism in the system 1) to issue the save command after detecting the fault (e.g., via input device 41).

[0087] If no fault is detected, the process 100 may return to operation 102, for example, to save additional robotic states of the robotic arm 4 to the state database 43, and / or to operation 108 to restore saved robotic states of the robotic arm 4 from the state database 43, until the surgical procedure is complete. However, if a fault is detected, at operation 106 the processor may execute to perform a recovery process to recover from the fault. This may include performing a specific recovery algorithm based on the type of fault. In some cases, this may include the robotic arm 4 being taken down or dismantled (moving them away from the patient and / or trocars, and / or removing surgical tools, e.g., a tear down from the robotic state, and returning to a stowed away configuration) to complete the recovery. In some cases, this may include guiding the robotic arm 4 back to a starting position or a safe predetermined position (e.g., determining the saved robotic state to be a safe orientation and position).

[0088] At operation 108, the processor can execute to receive a restore command to move the robotic arm from a second robotic state (e.g., a stowed away configuration, such as after recovering from a fault) back to the first robotic state. The restore command may be received for a stage in which a surgical procedure is repeated or following the recovery. For example, the surgical procedure may be repeated in a same surgery (e.g., continuing the surgical procedure after recovering from a fault, or repeating the surgical procedure in a next step) or a future surgery (e.g., utilizing the same surgical procedure for the patient or a next patient).

[0089] For example, the restore command may be input to restore the position and the orientation of the robotic arm 4 to a previously saved (pre -faulted) robotic state from the state database 43. This may include a command to drive the robotic arm 4 to the first position utilizing the surgical tool based on the state information saved in a data structure. The first robotic state can include a first orientation and a first position of the robotic arm 4, including a pose to be utilized to perform a surgical procedure (e.g., the robotic arm 4 may be coupled to a surgical tool, such as scissors, grasping jaws, endoscope, stapler, or camera, via a tool drive attached to a trocar). In some cases, the processor can issue the restore command automatically in response to recovering from a fault (without receiving human input from anDocket No. AUR6358WOPCT1 input device). In some cases, the processor can prompt the operator to provide the restore command.

[0090] At operation 110, in response to receiving the restore command, the processor can execute to determine a trajectory to the first robotic state. For example, the trajectory planner 46 can determine the trajectory to be commanded, based on the previously saved, first robotic state in the state database 43. In some cases, the processor can execute to simulate the robotic arm 4 moving along the trajectory to the first robotic state before driving the robotic arm 4 to avoid collisions with objects and / or other robotic arms 4. The simulated trajectory can be presented via a screen to the operator for approval.

[0091] At operation 112, the processor can execute to drive the robotic arm 4 from the second robotic state back to the first robotic state in response to the restore command. The robotic arm 4 may be driven to restore the position and the orientation (saved at operation 102 or operation 104) utilizing the surgical tool based on the state information (e.g., the first robotic state of the robotic arm 4, and in some cases, the type of surgical procedure, type of surgical tool, and / or the profile of the operator). In some cases, the robotic arm 4 may be driven to the first robotic state automatically, without receiving human input from an input device. In some cases, the robotic arm 4 may be driven to the first robotic state based on user input.

[0092] For example, the processor (e.g., utilizing the control signal generator 47) can drive the robotic arm 4 to the first position based on the determined trajectory to restore the position and the orientation. In some cases, the processor can drive the robotic arm 4 to the first position based on the simulated trajectory. In some cases, the processor can utilize sensing to sense an object and / or other robotic arm 4 along the trajectory to the first robotic state, to enable changing the trajectory to avoid collisions with objects and / or other robotic arm 4. For example, the processor can utilize sensors 44 to perform the sensing. The processor can execute to change the trajectory and / or stop the robotic arm 4 based on the sensing. In some cases, the processor can execute to drive the robotic arm 4 to an offset (e.g., the threshold distance D, above or outside of a saved robotic state) relative to an anatomical target or a patient or a trocar. The operator can then correct for any case-to-case variation.

[0093] At operation 114, the processor can resume operation of the robotic arm 4 to perform the surgical procedure (e.g., teleoperation, such as to perform the gastrectomy, a gastric bypass, a cholecystectomy, an orthopedic surgery, etc.). Additionally, the processor can execute to return to operation 102, for example, to save additional poses of the robotic arm 4 to the state database 43, and / or to operation 108 to restore saved positions andDocket No. AUR6358WOPCT1 orientations of the robotic arm 4 from the state database 43, until the surgical procedure is complete.

[0094] FIG. 11 is an example of a process 130 for performing surgical procedure utilizing the surgical robotic system of FIG. 1. At operation 132, a processor (e.g., one or more processors executing instructions stored in memory, such as a processor of the controller 40) in the surgical robotic system 1 can operate a robotic arm 4 (e.g., one or more robotic arms, such as four robotic arms) to perform a surgical procedure. For example, the robotic arm 4 may be controlled by an operator via teleoperation, such as to perform a gastrectomy, a gastric bypass, a cholecystectomy, an orthopedic surgery, etc. The robotic arm 4 may be driven to a first robotic state to perform a surgical procedure with a surgical tool 28, such as a starting configuration for a surgical procedure. For example, the first robotic state may enable the operator to reach certain anatomy in the body of a patient, via the surgical tool 28 coupled to a tool drive attached to a trocar (e.g., FIG. 9).

[0095] At operation 134, the processor can receive a save command to save the first robotic state of the robotic arm 4. The first robotic state may include a first orientation and a first position of the robotic arm 4 which may be utilizing the surgical tool 28 to perform the surgical procedure. For example, the surgical procedure could be an orthopedic surgery anchoring tissue to a bone. The first robotic state may be saved at a particular configuration in which tissue is pulled to a location corresponding to anchor spot to the bone.

[0096] The processor can save first state information in response to the save command. The first state information may indicate the first orientation and the first position. For example, the first orientation and the first position may be at a location of a trocar (e.g., trocar 77A, as shown in FIG. 9) or anatomy of a patient (e.g., the patient 6). In some cases, the first robotic state may include the position and orientation of the end effector 30 of the surgical tool 28.

[0097] At operation 136, the processor can drive from the first robotic state to a second robotic state in response to receiving a drive command. For example, the second robotic state may include a second orientation and a second position of the robotic arm 4. The second robotic state may correspond to a next step of the surgical procedure, such as moving away from the trocar or the anatomy of the patient, e.g., to zoom out with a camera, or moving to a next trocar or a next location of the anatomy. For example, for the orthopedic surgery, this could correspond to moving to a next trocar or location of the anatomy corresponding to a next anchor spot to attach tissue to the bone.

[0098] At operation 138, the processor can receive a restore command to move, from the second robotic state, back to the first robotic state. For example, the restore command may beDocket No. AUR6358WOPCT1 input to restore the position and the orientation of the robotic arm 4 to the previously saved first robotic state.

[0099] At operation 140, the processor can drive the robotic arm 4 to the first robotic state in response to receiving the restore command. The robotic arm 4 can be driven to restore the first orientation and the first position at the location of the trocar or anatomy of the patient, utilizing the surgical tool 28, based on the first state information. For example, for the orthopedic surgery, this could correspond to restoring the configuration of arms 4 to return to the location corresponding to the anchor spot to attach the tissue to the bone. The processor can continue to drive to other orientations and positions, save robotic states of those orientations and positions, and restore those orientations and positions based on the saved robotic states throughout the surgical procedure. Further, the processor can restore the orientations and positions of the saved robotic states in future surgical procedures.

[0100] Some aspects may perform variations to the process 100 described herein. For example, the specific operations of at least some of the processes might not be performed in the exact order shown and described. The specific operations might not be performed in one continuous series of operations, and different specific operations might be performed in different aspects.

[0101] As previously explained, an aspect of the disclosure may be a non-transitory machine-readable medium (such as microelectronic memory) having stored thereon instructions, which program one or more data processing components (generically referred to here as a “processor”) to automatically perform robot-assisted setup operations, as described herein. In other aspects, some of these operations might be performed by specific hardware components that contain hardwired logic. Those operations might alternatively be performed by any combination of programmed data processing components and fixed hardwired circuit components. A “processor” may include a distributed arrangement where multiple microprocessors are configured and controlled to perform the recited operations or tasks together, e.g., one processor can perform some of the recited operations and another processor can perform others of the recited operations. In some implementations, one or more aspects of various embodiments disclosed herein may be combined to form another embodiment.

[0102] While certain aspects have been described and shown in the accompanying drawings, it is to be understood that such aspects are merely illustrative of and not restrictive on the broad disclosure, and that the disclosure is not limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those ofDocket No. AUR6358WOPCT1 ordinary skill in the art. The description is thus to be regarded as illustrative instead of limiting.

[0103] In some aspects, this disclosure may include the language, for example, “at least one of [element A] and [element B] ” This language may refer to one or more of the elements. For example, “at least one of A and B” may refer to “A,” “B,” or “A and B.” Specifically, “at least one of A and B” may refer to “at least one of A and at least one of B,” or “at least of either A or B.” In some aspects, this disclosure may include the language, for example, “[element A], [element B], and / or [element C].” This language may refer to either of the elements or any combination thereof. For instance, “A, B, and / or C” may refer to “A,” “B,” “C,” “A and B,” “A and C,” “B and C,” or “A, B, and C.”

Claims

Docket No. AUR6358WOPCT1CLAIMSWhat is claimed is:

1. A method performed by a surgical robotic system, comprising:receiving a save command to save a first robotic state of a robotic arm of a surgical robotic system, the first robotic state including a first orientation and a first position of the robotic arm;saving first state information in response to the save command, wherein the first state information includes the first orientation and the first position of the robotic arm;driving the robotic arm from the first robotic state to a second robotic state in response to a drive command, the second robotic state being different than the first robotic state and including a second orientation and a second position;receiving a restore command to move the robotic arm from the second robotic state to the first robotic state; anddriving the robotic arm to the first robotic state in response to the restore command, wherein the robotic arm is driven to restore the first orientation and the first position based on the first state information.

2. The method of claim 1, further comprising:upon receiving the restore command, automatically repositioning the robotic arm to the first robotic state such that the robotic arm is autonomously guided from the second position and the second orientation to the first position and the first orientation to enable operation of a surgical tool coupled to the robotic arm.

3. The method of any one of claims 1-2, wherein, in response to receiving the restore command, the robotic arm is driven from the second robotic state to the first robotic state without receiving human input from an input device.

4. The method of any one of claims 1-3, wherein the first robotic state corresponds to a starting configuration for a surgical procedure.Docket No. AUR6358WOPCT1 5. The method of any one of claims 1-4, wherein the robotic arm is driven to the first robotic state based on determining the first robotic state to be a safe orientation and position.

6. The method of any one of claims 1-5, wherein the first robotic state results in a surgical tool being a first distance from an anatomical target and the second robotic state results in the surgical tool being a second distance from the anatomical target, the second distance being different than the first distance.

7. The method of any one of claims 1-6, wherein the restore command corresponds to a stage in which a surgical procedure is repeated.

8. The method of any one of claims 1-7, wherein the robotic arm includes an indicator to provide an indication after the robotic arm is driven to a position.

9. The method of claim 8, wherein the indicator provides a visual indication.

10. The method of claim 9, wherein the indicator comprises an LED ring that is at least partially circumferentially disposed around the robotic arm.

11. The method of claim 9, wherein the visual indication is output to a display.

12. The method of any one of claims 1-11, further comprising:simulating the robotic arm moving along a trajectory to the first robotic state before driving the robotic arm to avoid a collision with an object.

13. The method of any one of claims 1-12, further comprising:sensing an object along a trajectory to the first robotic state before driving the robotic arm to avoid a collision with the object.

14. The method of any one of claims 1-13, wherein the save command is generated via an input device utilized by an operator of the surgical robotic system.Docket No. AUR6358WOPCT1 15. The method of any one of claims 1-14, wherein the save command is automatically generated by the surgical robotic system.

16. The method of any one of claims 1-15, wherein the save command is generated in response to detecting a fault in the surgical robotic system.

17. The method of any one of claims 1-16, wherein the robotic arm is dismantled between receiving the save command and receiving the restore command.

18. The method of any one of claims 1-17, further comprising:outputting a prompt to an operator to issue the save command after detecting a preference of the operator or a fault in the surgical robotic system.

19. The method of any one of claims 1-18, wherein the first state information corresponds to a profile of an operator of the surgical robotic system or a type of surgical procedure to be performed.

20. The method of any one of claims 1-19, wherein the first state information indicates a configuration of a surgical tool coupled to a tool drive of the robotic arm.

21. The method of any one of claims 1-20, wherein the first robotic state is offset relative to a trocar or a patient of a surgical procedure.

22. A surgical robotic system, comprising:a robotic arm; anda processor configured to:receive a save command to save a first robotic state of the robotic arm of a surgical robotic system, the first robotic state including a first orientation and a first position of the robotic arm;save first state information in response to the save command, wherein the first state information includes the first orientation and the first position of the robotic arm;drive the robotic arm from the first robotic state to a second robotic state in response to a drive command, the second robotic state being different than the first robotic state and including a second orientation and a second position;Docket No. AUR6358WOPCT1 receive a restore command to move the robotic arm from the second robotic state to the first robotic state; anddriving the robotic arm to the first robotic state in response to the restore command, wherein the robotic arm is driven to restore the first orientation and the first position based on the first state information.

23. The surgical robotic system of claim 22, wherein a surgical tool is coupled to a tool drive of the robotic arm.

24. The surgical robotic system of any one of claims 22-23, wherein the processor is configured to:upon receiving the restore command, automatically reposition the robotic arm to the first robotic state such that the robotic arm is autonomously guided from the second position and the second orientation to the first position and first orientation to enable operation of a surgical tool coupled to the robotic arm.

25. The surgical robotic system of any one of claims 22-24, wherein, in response to receiving the restore command, the robotic arm is driven from the second robotic state to the first robotic state without receiving human input from an input device.

26. The surgical robotic system of any one of claims 22-25, wherein the first robotic state corresponds to a starting configuration for a surgical procedure.

27. The surgical robotic system of any one of claims 22-26, wherein the robotic arm is driven to the first robotic state based on determining the first robotic state to be a safe orientation and position.

28. The surgical robotic system of any one of claims 22-27, wherein the first robotic state results in a surgical tool being a first distance from an anatomical target and the second robotic state results in the surgical tool being a second distance from the anatomical target, the second distance being different than the first distance.

29. A method performed by a surgical robotic system, comprising:Docket No. AUR6358WOPCT1 receiving a save command to save a first robotic state of a robotic arm of a surgical robotic system, the first robotic state corresponding to a starting configuration for a surgical procedure, the first robotic state including a first orientation and a first position of the robotic arm, wherein the robotic arm utilizes a surgical tool to perform the surgical procedure, and the first robotic state results in the surgical tool being a first distance from an anatomical target;saving first state information in response to the save command, wherein the first state information includes the first orientation and the first position of the robotic arm;driving the robotic arm from the first robotic state to a second robotic state in response to a drive command, the second robotic state being different than the first robotic state and including a second orientation and a second position, wherein the second robotic state results in the surgical tool being a second distance from the anatomical target, the second distance being different than the first distance;receiving a restore command to move the robotic arm from the second robotic state to the first robotic state, the restore command corresponding to a stage in which the surgical procedure is repeated; anddriving the robotic arm to the first robotic state in response to the restore command, wherein the robotic arm is autonomously guided, upon receiving the restore command, from the second position and the second orientation to the first position and first orientation based on the first state information.

30. The method of claim 29, wherein, in response to receiving the restore command, the robotic arm is driven from the second robotic state to the first robotic state without receiving human input from an input device.

31. The method of any one of claims 29-30, wherein the robotic arm is driven to the first robotic state based on determining the first robotic state to be a safe orientation and position.

32. The method of any one of claims 29-31, wherein the robotic arm includes an indicator to provide a visual indication after the robotic arm is driven to a position.

33. The method of any one of claims 29-32, further comprising:Docket No. AUR6358WOPCT1 sensing an object along a trajectory to the first robotic state before driving the robotic arm to avoid a collision with the object.

34. The method of any one of claims 29-33, further comprising:outputting a prompt to an operator to issue the save command after detecting a preference of the operator or a fault in the surgical robotic system.

35. The method of any one of claims 29-34, wherein the first state information corresponds to a profile of an operator of the surgical robotic system or a type of surgical procedure to be performed.

36. The method of any one of claims 29-35, wherein the first state information indicates a configuration of the surgical tool coupled to a tool drive of the robotic arm.

37. The method of any one of claims 29-36, wherein the first robotic state is offset relative to a trocar or a patient of the surgical procedure.

Images