Detection of contact between components of manipulator arm assemblies

Abstract

A computer-assisted system includes a first manipulator arm assembly with a first manipulator arm, the first manipulator arm configured to support a first instrument, a second manipulator arm assembly with a second manipulator arm, the second manipulator arm configured to support a second instrument, first and second impact sensors disposed on the first and second manipulator arm assemblies, respectively, and a control system. The control system detects, within a detection time window, a first impact signature based on a first sensor output received from the first impact sensor, and a second impact signature based on a second sensor output received from the second impact sensor, and determines that a contact condition is present between the first manipulator arm assembly and the second manipulator arm assembly, based on the detection of the first impact signature and the detection of the second impact signature within the detection time window.

Description

DETECTION OF CONTACT BETWEEN COMPONENTS OF MANIPULATOR ARM ASSEMBLIES CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to and the benefit of the filing date of U.S. Provisional Patent Application 63 / 740,510, filed on December 31, 2024, which is hereby incorporated by reference herein in its entirety.BACKGROUNDField of Invention

[0002] The present invention generally provides improved computer-assisted systems and methods.Overview

[0003] Computer-assisted systems can be used to perform a task at a worksite.Example computer-assisted systems include industrial and recreational robotic systems. Example computer-assisted systems also include medical robotic systems used in procedures for diagnosis, non-surgical treatment, surgical treatment, etc. The computer-assisted system may include one or more manipulator arms to manipulate instruments. An instrument may be an instrument for performing the task. An instrument may also be an instrument for viewing the performing of the task. The computer-assisted system may be equipped with any number of instruments of any type.

[0004] During use of the computer-assisted system, one or more of the instruments and the manipulator arms supporting the instruments may be moved. For example, a manipulator arm with an instrument may be moved to interact with the worksite in different regions.

[0005] Contact between components of the manipulator arms and or the instruments may occur as a result of the movement. Improved techniques for operating computer-assisted systems that enable detection of a contact condition are, thus, desirable.SUMMARY

[0006] In general, in one aspect, one or more embodiments relate to a computer-assisted system comprising: a first manipulator arm assembly comprising a first manipulator arm, the first manipulator arm configured to support a first instrument; a second manipulator arm assembly comprising a second manipulator arm, the second manipulator arm configured to support a second instrument; a first impact sensor disposed on the first manipulator armassembly; a second impact sensor disposed on the second manipulator arm assembly; and a control system configured to: detect, within a detection time window, a first impact signature based on a first sensor output received from the first impact sensor; detect, within the detection time window, a second impact signature based on a second sensor output received from the second impact sensor; and determine that a contact condition is present between the first manipulator arm assembly and the second manipulator arm assembly, based on the detection of the first impact signature and the detection of the second impact signature within the detection time window.

[0007] In general, in one aspect, one or more embodiments relate to a computer-assisted system comprising: a first manipulator arm assembly comprising a first manipulator arm, the first manipulator arm configured to support a first instrument; a second manipulator arm assembly comprising a second manipulator arm, the second manipulator arm configured to support a second instrument; and a control system configured to: determine a contact force between the first manipulator arm assembly and the second manipulator arm assembly based on a tracking error in at least one of a plurality of joints of the first and the second manipulator arm; and determine that a contact condition is present between the first manipulator arm assembly and the second manipulator arm assembly, based on the contact force exceeding a pre-specified threshold.

[0008] In general, in one aspect, one or more embodiments relate to a method of operating a computer-assisted system, the computer-assisted system comprising: a first manipulator arm assembly comprising a first manipulator arm, the first manipulator arm configured to support a first instrument; a second manipulator arm assembly comprising a second manipulator arm, the second manipulator arm configured to support a second instrument; a first impact sensor disposed on the first manipulator arm assembly; a second impact sensor disposed on the second manipulator arm assembly; and the method comprising: detecting, within a detection time window, a first impact signature based on a first sensor output received from the first impact sensor; detecting, within the detection time window, a second impact signature based on a second sensor output received from the second impact sensor; and determining that a contact condition is present between the first manipulator arm assembly and the second manipulator arm assembly, based on the detection of the first impact signature and the detection of the second impact signature within the detection time window.

[0009] In general, in one aspect, one or more embodiments relate to a method of operating a computer-assisted system, the computer-assisted system comprising: a firstmanipulator arm assembly comprising a first manipulator arm, the first manipulator arm configured to support a first instrument; a second manipulator arm assembly comprising a second manipulator arm, the second manipulator arm configured to support a second instrument; and the method comprising: determining a contact force between the first manipulator arm assembly and the second manipulator arm assembly based on a tracking error in at least one of a plurality of joints of the first and the second manipulator arm; and determining that a contact condition is present between the first manipulator arm assembly and the second manipulator arm assembly, based on the contact force exceeding a prespecified threshold.

[0010] Other aspects of the invention will be apparent from the following description and the appended claims.BRIEF DESCRIPTION OF DRAWINGS

[0011] The accompanying drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component that is illustrated in various figures may be represented by a like numeral. For purposes of clarity, not every component may be labeled in every drawing. In the drawings:

[0012] FIG. 1 shows an example computer-assisted system in accordance with one or more embodiments.

[0013] FIG. 2A shows a scenario involving a contact condition in accordance with one or more embodiments.

[0014] FIG. 2B shows a scenario involving a contact condition in accordance with one or more embodiments.

[0015] FIG. 3 is a diagram of a system for detecting contact conditions in accordance with one or more embodiments.

[0016] FIG. 4 shows a flowchart describing an example method for detecting contact conditions in accordance with one or more embodiments.

[0017] FIG. 5 shows an example of the execution of a method in accordance with one or more embodiments.DETAILED DESCRIPTION

[0018] Specific embodiments of the disclosure will now be described in detail with reference to the accompanying figures. Like elements in the various figures are denoted by like reference numerals for consistency.

[0019] In the following detailed description of embodiments of the disclosure, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to one of ordinary skill in the art that the invention may be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the description.

[0020] Throughout the application, ordinal numbers (e.g., first, second, third, etc.) may be used as an adjective for an element (i.e., any noun in the application). The use of ordinal numbers is not to imply or create any particular ordering of the elements, and is not to limit any element to being only a single element unless expressly disclosed, such as by the use of the terms “before”, “after”, “single”, and other such terminology. Rather, the use of ordinal numbers is to distinguish between the elements. By way of an example, a first element is distinct from a second element, and the first element may encompass more than one element and succeed (or precede) the second element in an ordering of elements.

[0021] This disclosure describes various devices, elements, and portions of computer-assisted systems and elements in terms of their state in three-dimensional space. As used herein, the term “position” refers to the location of an element or a portion of an element (e.g., three degrees of translational freedom in a three-dimensional space, such as along Cartesian x-, y-, and z-coordinates). As used herein, the term “orientation” refers to the rotational placement of an element or a portion of an element (e.g., three degrees of rotational freedom in three-dimensional space, such as about roll, pitch, and yaw axes, represented in angle-axis, rotation matrix, quaternion representation, and / or the like). As used herein, and for a device with a kinematic series, such as with a repositionable structure with a plurality of links coupled by one or more joints, the term “proximal” refers to a direction toward a base of the kinematic series, and “distal” refers to a direction away from the base along the kinematic series.

[0022] As used herein, the term “pose” refers to the multi-degree of freedom (DOF) spatial position and orientation of a coordinate system of interest attached to a rigid body. In general, a pose includes a pose variable for each of the DOFs in the pose. For example, a full 6-DOF pose for a rigid body in three-dimensional space would include 6 pose variables corresponding to the 3 positional DOFs (e.g., x, y, and z) and the 3 orientational DOFs (e.g., roll, pitch, and yaw). A 3-DOF position only pose would include only pose variables for the 3 positional DOFs. Similarly, a 3-DOF orientation only pose would include only pose variables for the 3 rotational DOFs. Further, a velocity of the pose captures the change inpose over time (e.g., a first derivative of the pose). For a full 6-DOF pose of a rigid body in three-dimensional space, the velocity would include 3 translational velocities and 3 rotational velocities. Poses with other numbers of DOFs would have a corresponding number of velocities translational and / or rotational velocities.

[0023] Aspects of this disclosure are described in reference to computer-assisted systems, which can include devices that are teleoperated, externally manipulated, autonomous, semiautonomous, and / or the like. Further, aspects of this disclosure are described in terms of an implementation using a teleoperated surgical system, such as the da Vinci® Surgical System commercialized by Intuitive Surgical, Inc. of Sunnyvale, California. Knowledgeable persons will understand, however, that inventive aspects disclosed herein may be embodied and implemented in various ways, including teleoperated and nonteleoperated, and medical and non-medical embodiments and implementations.Implementations on da Vinci® Surgical Systems are merely exemplary and are not to be considered as limiting the scope of the inventive aspects disclosed herein. For example, techniques described with reference to surgical instruments and surgical methods may be used in other contexts. Thus, the instruments, systems, and methods described herein may be used for humans, animals, portions of human or animal anatomy, industrial systems, general robotic, or teleoperated systems. As further examples, the instruments, systems, and methods described herein may be used for non-medical purposes including industrial uses, general robotic uses, sensing or manipulating non-tissue work pieces, cosmetic improvements, imaging of human or animal anatomy, gathering data from human or animal anatomy, setting up or taking down systems, training medical or non-medical personnel, and / or the like.Additional example applications include use for procedures on tissue removed from human or animal anatomies (with or without return to a human or animal anatomy) and for procedures on human or animal cadavers. Further, these techniques can also be used for medical treatment or diagnosis procedures that include, or do not include, surgical aspects.

[0024] Referring now to the drawings, in which like reference numerals represent like parts throughout the several views, FIG. 1 shows an example computer-assisted system 100, in accordance with one or more embodiments. The computer-assisted system includes a repositionable assembly 110.

[0025] In some embodiments, the repositionable assembly 110 (e.g., a patient-side robotic device in a medical example) includes one or more manipulator arm(s) 150, each of which may support a removably coupled instrument 160 (also called tool 160). A manipulator arm assembly 130 comprises a manipulator arm 150. When an instrument 160 isinstalled on, coupled to, or otherwise supported by a manipulator arm 150, the combination of the manipulator arm 150 and the instrument 160 forms a manipulator arm assembly 130. The computer-assisted system 100 may include a drivable cart (not shown) with a primary control interface and a support column that vertically extends from a drivable cart. In some embodiments, a boom extends from the support column with at least one manipulator arm 150 connected to the distal end of the boom. In the example of FIG. 1, the repositionable assembly 110 is depicted with four manipulator arms 150, namely, a first manipulator arm 150A, a second manipulator arm 150B, a third manipulator arm 150C, and a fourth manipulator arm 150D. With instruments installed on these manipulator arms, the repositionable assembly 110, thus, includes the manipulator arm assemblies 130A, 130B, 130C, and 130D. Where a particular manipulator arm need not be referenced, a manipulator arm may be referenced generally as manipulator arm 150. Similarly, where a particular manipulator arm assembly need not be referenced, a manipulator arm assembly may be referenced generally as manipulator arm assembly 130.

[0026] In this example, the repositionable assembly 110 is utilized for a diagnostic or therapeutic medical procedure performed on a patient P on an operating table. The instrument 160 may enter the workspace through an entry location (e.g, enter the body of the patient P through a natural orifice such as the throat or anus, or through an incision), while an operator (not shown) views the worksite (e.g., a surgical site in the surgical scenario) through a user interface system.

[0027] An image of the worksite may be obtained by an imaging instrument 160 including an imaging device (e.g., an endoscope, an optical camera, an ultrasonic probe, etc. in a medical example). The imaging instrument 160 can be used for imaging the worksite, and may be manipulated by one of the manipulator arms 150A-D of the repositionable assembly 110 so as to position and orient the imaging instrument 160.

[0028] The number of instruments 160 used at one time generally depends on the task and space constraints, among other factors. If it is appropriate to change, clean, inspect, or reload one or more of the instruments 160 being used during a procedure, an assistant (not shown) may remove the instrument 160 from the manipulator arm 150A-D, and replace it with the same instrument 160 or another instrument 160.

[0029] Various classes and types of instruments 160 can be used, or exchanged during a process or procedure, according to the needs of a procedure or process undertaken by one or more manipulator arms (e.g, 150A-D) of a computer-assisted system 100.Instruments 160 may include, but are not limited to: monopolar and bipolar instruments fordissection, coagulation, sealing, transection, and / or cutting of tissue; suction and irrigation instruments; clip appliers; and instruments for visualization of the procedure site (e.g., visual camera, infrared camera). Some instruments 160, such a monopolar and bipolar instruments, can be “energized” or apply electrical cortical stimulation in a tunable manner (e.g., “on” or “off’).

[0030] While FIG. 1 shows the computer-assisted system 100 as a medical system, the following description is applicable to other scenarios and systems (e.g., medical scenarios or systems that are non-surgical, non-medical scenarios or systems, etc.).

[0031] Furthermore, while example configurations of computer-assisted systems 100 including repositionable assemblies 110 are shown in FIG. 1, those skilled in the art will appreciate that embodiments of the disclosure can be used with any design of manipulator structure. For example, a repositionable assembly 110 can have any number and any types of degrees of freedom and / or have a configuration different from what is shown in FIG. 1. A repositionable assembly may further include multiple manipulator arm assemblies with no common support, e.g., mounted on separate carts, or with a common support different from what is illustrated in FIG. 1. For example, the common support may be formed by ceilingmounted or table-mounted elements such as rails, gantries, or other structures.

[0032] The computer-assisted system 100 further includes a control system 170.

[0033] The illustration, partitioning, organization, and interaction of the components and / or modules of the control system 170 is intended to promote clear discussion and should not be considered fixed or limiting. For example, FIG. 1 depicts a display unit 174 as an independent entity, however, it is well understood that in practice the display unit 174 may be implemented as part of, for example, a user input system 140 and / or an auxiliary system 180.

[0034] The control system 170 includes one or more computing systems 172. A computing system 172 includes a processing system and is used to process input provided by the user input system 140 (e.g., from input device(s) manipulated by an operator). In some embodiments, a computing system 172 is further used to provide an output (e.g., a video, an image) to the display unit 174. Examples of display unit 174 include LCDs, LEDs, organic LED displays, projectors, etc. The display unit 174 may be designed of monoscope and / or stereoscopic operation. In some embodiments, one or more computing systems 172 are used to control the repositionable assembly 110.

[0035] In one or more embodiments, the computing system(s) 172 executes one or more control methods. The control methods include instructions for controlling one or more components of a repositionable assembly 110.

[0036] In one or more embodiments, joint movements of the repositionable assembly are controlled by one or more control methods driving one or more joints using actuators of the repositionable assembly, the joint movements being calculated by a processor of a processing system of the computing system(s) 172. The control methods process control signals from the user input system 140 or elsewhere, and / or sensor signals (e.g., positional encoder data from joint position sensors, image data from image instruments such as ultrasonic probes or cameras or endoscopes, and / or the like), to calculate commands for the joint actuators.

[0037] In some embodiments, the control methods perform at least some of the calculations of the joint commands using vectors and / or matrices, some of which have elements corresponding to positions, velocities, and / or forces / torques of the joints. The range of alternative joint configurations available to the control methods can be conceptualized as a joint space. For example, in some embodiments, the joint space has as many dimensions as the repositionable assembly has degrees of freedom, and a particular configuration of the repositionable assembly represents a particular point in the joint space, with each coordinate corresponding to a joint state of an associated joint of the repositionable assembly.

[0038] As used herein, the term “state” of a joint or multiple joints refers to the control variables associated with the joint or the multiple joints, respectively. For example, the state of an angular joint refers to the angle defined by that joint within its range of motion, and / or to the angular velocity (or speed or direction) of the joint. Similarly, the state of an axial or prismatic joint refers to the joint's axial or linear position, and / or to its axial or linear velocity (or speed or direction). While one or more of the control methods described herein include position controllers, they often also have velocity control aspects. Alternative embodiments can rely primarily or entirely on velocity controllers, force controllers, acceleration controllers, and / or the like without departing from the disclosure. Various aspects of control systems that can be used in such devices are described in U.S. Patent No.6,699,177, which is incorporated herein by reference. In general, as long as the movements described are based on the associated calculations, the calculations of movements of the joints and movements of an end effector described herein are performed using a position control technique, a velocity control technique, an acceleration control technique, a force or torque control technique, a combination of some or all of the foregoing, and / or the like.

[0039] In some embodiments, the control modes include one or more other types of control modes. For example, during a robotic task being performed under the control of input devices operated by a user, various joints of the repositionable assembly can be commandedto a same position and controlled to maintain static positions. However, in another control mode, one or more of the joints can be commanded to be “floating,” and facilitate motion of that joint due to externally applied force. For example, a joint held in place by a brake can be floated by partially or entirely releasing the brake. As another example, a joint driven by actuator(s) can be held in place by commanding the actuator(s) to hold the joint position, and be floating by updating the command to the actuator(s) to the then-current position, velocity, and / or acceleration of the joint.

[0040] In various embodiments, multiple different control modes are combined during operation of the repositionable assembly. For example, some joints could be controlled to maintain position and resist or rebound from attempted external articulation of those joints, while other joints could be controlled to move those other joints. Parameters such as joint position, velocity, or acceleration of the joints are detected by joint sensors. Example sensors may include those associated with actuators or joints of the computer-assisted system, such as motor encoders, rotary or linear joint encoders, torque sensors, current sensors, accelerometers, force sensors, inertial measurement units, optical or ultrasonic sensors or imagers, RF sensors, etc. The sensor signals are used to provide kinematic information of the repositionable assembly.

[0041] In one aspect, the computer-assisted system 100 may utilize one or more clutch modes in which one or more joints are allowed to float along one or more axes (e.g., one or more degrees-of-freedom) within a joint velocity sub-space that results in movement of a distal portion of the repositionable assembly, such as an end effector (e.g., instrument 160) or a remote center about which an instrument shaft of the end effector pivots adjacent a minimally invasive aperture during surgery. This joint velocity sub-space is referred to as a null-perpendicular space, which is orthogonal to a null-space in which movement of the joints results in no movement of a distal portion of manipulators assembly.

[0042] In many embodiments, the repositionable assembly comprises a surgical tool or instrument having a shaft extending between a mounting interface and an end effector. In a first operational mode, the control system 170 may be configured to derive the desired movement of the end effector within an internal surgical space so that the shaft passes through a minimally invasive aperture site (e.g., a port). A leader-follower controller may, for example, comprise a velocity controller using a Jacobian pseudo-inverse matrix.

[0043] The architecture of the control methods used for controlling the repositionable assembly can be of any appropriate form. As a specific example, the control architecture can be hierarchical, and could include a high-level controller and multiple joint controllers. Acommanded movement is received by the high-level controller in, for example, a Cartesiancoordinate space (referred to herein as Cartesian-space). The commanded movement could be, for example, based on a movement command (e.g., in the form of a position and / or velocity) received from the user input system 140, or any other system that provides a movement command. The commanded movement is converted into commanded joint positions or joint velocities (e.g., linear or angular joint positions, linear or angular joint velocities). In some embodiments, the conversion is performed using an inverse kinematics algorithm. Subsequently, the joint controllers convert the received commanded joint positions or velocities into commanded currents to drive the actuators producing joint movements. The joint movements together produce a repositionable assembly movement that reflects the commanded movement.

[0044] In some embodiments, a joint controller controls a joint position. In some embodiments, the joint controller controls other variables such as joint velocity and / or joint force (linear force or angular torque). A joint controller receives a feedback signal in the form of a sensed joint state from an associated joint sensor, which it can use for closed-loop control. The sensed joint state includes, for example and without limitation, a joint position, a joint velocity (or component of velocity such as speed or direction), a joint acceleration (or component of acceleration), and / or the like, representing the joint movement. The sensed joint state is derived from the signals obtained from the joint sensor. A joint sensor can be, for example, an encoder, a potentiometer, an accelerometer, a hall effect sensor, and / or the like. In some embodiments, a state observer or estimator (not shown) is used. Each joint controller can implement any appropriate control scheme, such as a proportional integral derivative (PID), proportional derivative (PD), full state feedback, sliding mode, and / or various other control schemes, without departing from the disclosure.

[0045] The computing system 172 may include, without limitation, one or more computer processors, non-persistent storage (e.g, volatile memory, such as random access memory (RAM), cache memory), persistent storage (e.g, a hard disk, an optical drive such as a compact disk (CD) drive or digital versatile disk (DVD) drive, a flash memory, etc.), a communication interface (e.g., Bluetooth interface, infrared interface, network interface, optical interface, etc.), and / or numerous other elements and functionalities. In some embodiments, a computer processor of the computing system 172 is an integrated circuit for processing instructions. For example, the computer processor can be one or more cores or micro-cores of a processor.

[0046] In some embodiments, a communication interface of a computing system 172 includes an integrated circuit for connecting the computing system 172 to the repositionable assembly 110, a network (not shown) (e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, mobile network, or any other type of network), and / or to another device, such as another computing system 172.

[0047] Further, in some embodiments, the computing system 172 includes one or more output devices, such as a display unit 174, a printer, a speaker, external storage, or any other output device. A display unit 174 may be a liquid crystal display (LCD), a plasma display, projector, headset device, touchscreen, organic LED display (OLED), projector, or other display device. Output devices may include, but are not limited to, a printer, a speaker, external storage, or any other output device. One or more of the output devices may be the same or different from the input device(s). Many different types of computing systems 172 exist, and the aforementioned input and output device(s) may take other forms.

[0048] Software instructions in the form of computer readable program code to perform embodiments of the disclosure may be stored, in whole or in part, temporarily or permanently, on a non-transitory computer readable medium (machine-readable medium) such as a CD, DVD, storage device, a diskette, a tape, flash memory, physical memory, or any other computer readable storage medium. Specifically, the software instructions may correspond to computer readable program code that, when executed by a processor(s), is configured to perform one or more embodiments of the disclosure.

[0049] The computing system 172 may be connected to or be a part of a network. The network may include multiple nodes. Each node may correspond to a control system, or a group of nodes. By way of an example, embodiments of the disclosure may be implemented on a node of a distributed system that is connected to other nodes. By way of another example, embodiments of the disclosure may be implemented on a distributed computing system having multiple nodes, where each portion of the disclosure may be located on a different node within the distributed computing system. Further, one or more elements of the aforementioned computing systems 172 may be located at a remote location and connected to the other elements over a network.

[0050] In some embodiments, the control system 170 includes one or more auxiliary system(s) 180. An auxiliary system 180 may be any system that operates in coordination with the repositionable assembly 110. For example, an auxiliary system 180 may process captured images from an imaging instrument 160. This non-limiting example of the auxiliary system 180 may include a variety of methods for processing captured images prior to anysubsequent display. For example, the auxiliary system 180 may overlay the captured images with a virtual control interface prior to displaying the combined images to the operator via the user input system 140 or other display units 174 located locally or remotely from the procedure. One or more separate displays 174 may also be coupled with a computing system 172 and / or the auxiliary system 180 for local and / or remote display of images, such as images of the procedure site, or other related images.

[0051] FIG. 2A shows a first scenario 200A involving a contact condition in accordance with embodiments of the disclosure. The first scenario is further described below, after an introduction of the various components potentially involved in the contact scenario.

[0052] The scenario in FIG. 2A involves two manipulator arms 250A, 250B. Each of the manipulator arms is configured to support an instrument. In the scenario as shown, manipulator arm 250A supports instrument 260A, and manipulator arm 250B supports instrument 260B. Manipulator arm 250A in combination with instrument 260A forms manipulator arm assembly 230A, and manipulator arm 250B in combination with instrument 260B forms manipulator arm assembly 230B.

[0053] In the example, the manipulator arms 250A, 250B and the instruments 260A, 260B are illustrated as essentially mirror-symmetric. However, the disclosure is not limited to manipulator arms and instruments of a particular design or kinematic configuration. The following introduced various components of manipulator arms and instruments and is based on manipulator arm 250A and instrument 260A. The description is equally applicable to manipulator arm 250B and instrument 260B.

[0054] Manipulator arm 250A and instrument 260A may correspond to, for example, one of the manipulator arms 150A, 150B, 150C, 150D in FIG. 1 with the corresponding instrument. During operation, the manipulator arm 250A generally supports the instrument 260A extending distally from the manipulator arm 250A, and effects movements of the instrument 260A.

[0055] In minimally invasive scenarios, an instrument 260 may be positioned and manipulated through an entry location 296 of a barrier 298 (e.g., an incision or a natural orifice of patient P), so that a kinematic remote center is maintained at the entry location. Images of the worksite, taken by an imaging device of an imaging instrument such as one comprising one or more optical cameras, may include images of the distal ends of the instruments 260 when the instruments 260 are positioned within the field-of-view of an imaging device. In other scenarios, including non-medical scenarios or medical trainingscenarios, the barrier 298 may be any kind of barrier that separates an internal volume (e.g., a workspace for instruments 260A and 260B) from an external environment.

[0056] In the example as shown, a distal instrument holder 214 facilitates removal and replacement of the mounted instrument.

[0057] As may be understood with reference to FIG. 1, manipulator arm 250A is supported by, coupled to, or otherwise a part of the repositionable assembly that supports additional manipulator arms such as manipulator arm 250B. Alternatively, different manipulator arms may be mounted to separate bases that may be independently movable, e.g., by the manipulator arm 250A being mounted to a single-manipulator-arm cart, being provided with mounting clamps that allow mounting of the manipulator arm 250A directly or indirectly to the operating table at various locations, etc. An example manipulator arm 250A includes a plurality of links and joints extending between the proximal base and the distal instrument holder.

[0058] In embodiments such as shown for example in FIG. 2A, a manipulator arm includes multiple joints (such as revolute joints JI, J2, J3, J4, and J5, and prismatic joint J6) and links (204, 206, 208, and 210). The joints of the manipulator arm, in combination, may or may not have redundant degrees of freedom.

[0059] In the example shown, the instrument 260A may be releasably mounted on an instrument holder 214. The instrument holder 214 may translate along a linear guide formed by the prismatic joint J6 at the link 210. Thus, the instrument holder 214 may provide in / out movement of the instrument 260A along an insertion axis. The link 210 may further support a cannula 216 through which the instrument shaft 218 of the instrument 260A extends.Alternatively, the cannula 216 may be mechanically supported by another component of the manipulator arm 250A or may not be supported at all.

[0060] Actuation of the instrument 260A in one or more degrees may be provided by actuators of the manipulator arm 250A. These actuators may be integrated in the instrument holder 214, or drive drivetrains in the instrument holder, and may actuate the instrument 260A via the transmission assembly 270.

[0061] According to the above description, the execution of a procedure, e.g., a medical procedure may involve not only movement of instrument 260A (e.g., opening, closing, articulating an end effector of the instrument), but also reconfiguration of the manipulator arm 250A, e.g., by actuating one or more of joints JI, J2, J3, J4, and J5, and J6. The execution of the procedure may further involve movement of instrument 260B and manipulator arm 250B. During such movements, components of manipulator arm assembly230 A and components of manipulator arm assembly 230B may come in contact. FIG. 2 A illustrates such a contact condition Cl.

[0062] The contact condition Cl as shown is a contact condition between two manipulator arms. In some examples such a contact condition occurs external to the barrier 298. Such a contact condition may be a result of a contact between any component of manipulator arm 250A and any component of manipulator arm 250B. The contact condition may also be a result of a contact between any component of instrument 260A that is external to the barrier 298 and a component of manipulator arm 250B or instrument 260B. In the example as illustrated, the contact condition Cl occurs between the transmission assemblies 270 which are components of instruments 260A and 260B. Other contact conditions Cl could occur between one of links 204, 206, 208, 210 of the manipulator arms 250A, 250B, or between one of the links and a component of an instrument, e.g. an element of the instrument back end, such as a transmission assembly 270.

[0063] FIG. 2B shows a second contact scenario 200B in accordance with embodiments of the disclosure. Analogous to the first contact scenario 200A, the second contact scenario 200B involves the two manipulator arms 250A, 250B that support instruments 260A, 260B, respectively.

[0064] In the second contact scenario 200B, as a result of movement of instrument 260A, manipulator arm 250A, instrument 260B, and / or manipulator arm 250B, components of instrument 260A and instrument 260B come in close proximity or even in contact. FIG.2B illustrates such a contact condition C2.

[0065] The contact condition C2 as shown is a contact condition between two instrument shafts. In some examples such a contact condition occurs internal to the barrier 298. Such a contact condition may be a result of a contact condition between any component of instrument 260 A and any component of instrument 260B, internal to the barrier 298. Examples of such contact conditions include contacts between instrument shafts 218 of instruments 260A, 260B, between end effectors 220 if instruments 260A, 260B, and between an end effector of one instrument and an instrument shaft of the other instrument.

[0066] While, as FIGs. 2A and 2B illustrate, different types of contact conditions may occur, these contact conditions have in common that they involve physical contact between components of a pair of manipulator arm assemblies (e.g., 230A, 230B). A contact condition may occur between any possible pair of manipulator arm assemblies. Referring to FIG. 1, in case of a repositionable assembly 110 with four manipulator arm assemblies 130A, 130B, 130C, 130D, the following pairs may be involved in a contact condition: 130A-130B, 130A-130C, 130A-130D, 130B-130C, 130B-130D, 130C-130D. A contact condition may occur anywhere in space between the two manipulator arm assemblies in a pair of manipulator arm assemblies. As FIGs. 2A and 2B distinguish, contact conditions may be separated based on whether they occur external to the barrier 298 (Cl in FIG. 2 A) and internal to the barrier 298 (C2 in FIG. 2B). While contact conditions external to the barrier may occur between manipulator arms, between instruments, and between manipulator arms and instruments, contact conditions internal to the barrier may occur only between instruments, as previously discussed.

[0067] Embodiments of the present disclosure contemplate determining or identifying characteristics of a collision based at least in part on sensor data. Different forms of contact conditions may further be distinguished based on how the contact condition occurs.

[0068] Contact conditions may be a relatively common occurrence and a user of the computer-assisted system (e.g., a surgeon) may not necessarily be aware of contact conditions when they occur.

[0069] Embodiments of the disclosure notify the user of a contact condition when the contact condition is detected. This avoids otherwise confusing situations and facilitates and streamlines the operation of the computer-assisted system by the user.

[0070] Consider, for example, a configuration that renders a force feedback to the input device operated by the user. In one example, the force (and / or torque) rendered to the input device is proportional to a joint tracking error in the manipulator arm assembly. Such a joint tracking error could be a result of a contact condition (assume, for example, that a manipulator arm cannot follow the user command because it is blocked by another manipulator arm). However, the joint tracking error could also be the result of a procedure-related force or torque (in absence of a contact condition). Embodiments of the disclosure make these two cases distinguishable, for example, by detecting and making the user aware of the presence of a contact condition. Embodiments of the disclosure may be particularly beneficial when a computer-assisted system is outfitted with force-feedback instruments. Forces (and / or torques) from the force-feedback instrument may be rendered to the user when interacting with the surrounding environment (e.g., when manipulating tissue in a surgical scenario). In such a configuration, the user may rely on the force-feedback at the input device to control movement of the instrument. In one or more embodiments, when a contact condition is detected, the user may be made aware of the contact condition, such that the user does not misinterpret the rendered force at the input device as feedback from the instrument when, in fact, the rendered force is caused by the contact condition. In other words,embodiments of the disclosure help disambiguate the source of force feedback. This, in turn, enables the user to react differently, depending on whether the force-feedback is associated with a force measured by a force-feedback instrument or a contact condition. Referring to FIG. 2A, which illustrates a contact condition in which two manipulator arms are in collision (which can occur external to a barrier), a user operating the computer-assisted system, in absence of embodiments of the disclosure, may be completely unaware of such a contact condition because the user may control movement of the instrument(s) based on a camerabased visual feedback obtain from the region internal to the barrier. In a medical scenario, such visual feedback may be obtained by an instrument equipped with an endoscopic camera. The user may not have a direct view of the manipulator arm assembly with its components external to the barrier. Referring to FIG. 2B which illustrates a contact condition between two instrument shafts (which can occur internal to the barrier), while camera-based visual feedback may be available from the region internal to the barrier, the camera (e.g., an endoscopic camera) is typically placed with an end effector(s) in the field of view of the camera, e.g., to allow visual supervision and control of the end effector as it is interacting with the surrounding environment. Certain contact conditions, e.g., between instrument shafts, may be outside the field of view of the camera. Accordingly, even inside the barrier, there may be contact conditions that are not visually perceivable by the user. Further such contact conditions may be particularly unpredictable, because elements of the instrument such as the instrument shaft may be flexible. In presence of a flexible element, movement may still be possible in presence of a contact condition, although the movement may be altered and / or limited. Further, when a contact condition is abruptly resolved (e.g., by one instrument shaft slipping past another blocking instrument shaft) the instrument shaft may abruptly snap back to its original configuration, thus causing unexpected motion that may surprise or startle the user. By reporting contact conditions to the users, the user is made aware of such contact conditions, enabling the user to respond in an appropriate manner. To further increase user awareness and / or to facilitate resolution of a contact condition, the location of the contact condition (e.g., a spatial location), the type of the contact condition (e.g., a first manipulator arm being in contact with a second manipulator arm vs a first instrument shaft being in contact with a second instrument shaft) and / or manipulator arm assemblies and / or links of the manipulator arm assemblies involved in the contact condition may be identified to the user. The operations performed to enable the detection and reporting of contact conditions are subsequently described.

[0071] Embodiments of the disclosure provide a detection of contact condition between manipulator arm assemblies, as introduced, e.g., in reference to FIGs. 2A and 2B.

[0072] Referring to FIG. 2A, some embodiments of the disclosure use signals obtained from impact sensors 222A, 222B to perform the detection. An impact sensor may be designed to detect impacts or contact conditions between manipulator arm assemblies. For example, impact sensor 222 A may detect an impact at manipulator arm assembly 230 A, and impact sensor 222B may detect an impact at manipulator arm assembly 230B. Each manipulator arm assembly of a repositionable assembly may be equipped with respective one or more impact sensor(s). For example, four impact sensors may be used in case of repositionable assembly 110.

[0073] An impact sensor may be placed anywhere on a manipulator arm assembly. In some embodiments, the impact sensor is placed such that it is more likely to detect commonly occurring impacts, such as contact conditions between manipulator arms. As illustrated in FIG. 2A, a distal link of a manipulator arm is more likely to be involved in a contact condition than a proximal link of the manipulator arm. In the example, the impact sensor 222A is disposed on link 210. In this configuration, the impact sensor may be able to detect contact conditions that occur at the distal links of the manipulator arm 250A, and further potentially contact conditions that involve components of the instrument 260A, e.g., at the instrument shaft 218. In other embodiments, the impact sensor can be a component of or be placed on the instrument, e.g., on the backend of the instrument at or near the transmission assembly 270.

[0074] The impact sensor may be any type of sensor capable of detecting an impact as described. In one or more embodiments, an inertial measurement unit (IMU) is used. An IMU may include accelerometers, gyroscopes, and / or magnetometers. Sensor data from these sensors may be fused to optimize detection. Alternatively, other sensors may be used, e.g., acceleration sensors. A more detailed description of the detection using impact sensors is provided below.

[0075] FIG. 3 is a diagram of a system 300 for detecting a contact condition in accordance with one or more embodiments. FIG. 3 illustrates the detection of a contact condition for two manipulator arm assemblies. An impact sensor A 222A is configured to detect impacts at a manipulator arm assembly A and an impact sensor B 222B is configured to detect impacts at a manipulator arm assembly B. The system 300 as shown may be extended to detect contact conditions between additional manipulator arm assemblies by adding additional mechanical impact sensors.

[0076] In one or more embodiments, a contact condition determination engine 310 is configured to detect a contact condition between the two manipulator arm assemblies. For example, the contact condition determination engine 310 is configured to detect contact conditions between manipulator arm assemblies A and B. To perform the detection, a method is performed as described in reference to FIG. 4.

[0077] Broadly speaking, the contact condition determination engine 310 receives sensor output A from impact sensor A 222A and a sensor output B from impact sensor B 222B. The contact condition determination engine 310 determines the presence of a contact condition depending on the presence of impact signatures on the sensor outputs.

[0078] An impact signature may be detected based on sensor output from an impact sensor, the impact signature indicating an impact on or affecting the impact sensor. Thus, detection of an impact signature based on sensor output from an impact sensor associated with a manipulator arm assembly may be indicative of an impact on or affecting the manipulator arm assembly. An impact signature may be a spike or a series of spikes on the corresponding sensor output. More generally, when mechanical bodies collide, a high frequency impact signature is registered by the impact sensors of the manipulator arm assemblies involved in the contact condition, thereby indicating that there is a collision between these two manipulator arm assemblies. An example is provided in FIG. 5.

[0079] If impact signatures are present on sensor output A and on sensor output B, within a detection time window, then the contact condition determination engine 310 may determine that the contact condition is present between the manipulator arm assemblies on which impact sensor A and impact sensor B are positioned (manipulator arm assemblies A and B in this example). The detection time window may be selected such that the mechanical impact signatures on sensor outputs A and B are temporally aligned within, for example, a few milliseconds to ensure that they can be attributed to the same mechanical contact event, thereby suggesting that the mechanical contact event is the result of a mechanical contact between manipulator arm assemblies A and B. More generally, the temporal alignment may use any method to determine the temporal alignment of the impact signatures on sensor outputs A and B, e.g., based on whether the time elapsed between the detections of the two impact signatures does / does not exceed a threshold time.

[0080] Some processing may be performed to facilitate the detection of an impact signature in the detection time window. Assume, for example, that the signal of one or more accelerometers of the IMU is processed. With accelerometers for multiple degrees of freedom (e.g., three degrees of freedom) being part of an IMU, the acceleration signal mayhave an amplitude and a direction. The amplitude may be used to detect the mechanical impact signature. To improve detection, other known acceleration components (e.g., gravity) may be removed, thereby providing an unbiased scalar signal representing the magnitude of acceleration attributable to or otherwise caused by the contact condition. The unbiased acceleration magnitude signal may then be filtered to isolate specific frequency components, e.g., components of the acceleration with frequencies above 100Hz. A high-pass filter may be used. For example, a sixth-order high-pass filter may be used. The resulting filtered signal may be indicative of contact conditions based on the presence of spikes that occur simultaneously or near-simultaneously on multiple sensor outputs obtained from the IMUs associated with the manipulator arm assemblies involved in the contact condition.

[0081] Alternative methods may be used to detect mechanical impact signatures on sensor outputs A and B, without departing from the disclosure, for example, statistical methods such as the Pearson correlation coefficient, can be used to determine a correlation between sensor outputs A and B. In such an implementation, a high correlation may be used as an indication for a contact condition.

[0082] A contact condition between manipulator arm assemblies A and B is determined by the contact condition determination engine 310 only if the collision signature is present on both sensor outputs A and B. The contact condition between arm assemblies A and B is not determined if the impact signature is detected only by impact sensor A or only by impact sensor B. Assume, for example, that the scenario as described further includes a manipulator arm assembly C equipped with an impact sensor C. Further assume that no impact signature is detected based on sensor output C, while both sensor outputs A and B carry the impact signature. Based on this pattern, the contact condition determination engine 310 may conclude that the contact condition is between manipulator arm assemblies A and B, but does not involve manipulator arm assembly C.

[0083] Continuing with the discussion of the contact condition determination engine 310, the contact condition determination engine 310 may also receive an input representing a tracking error. The tracking error may be obtained from control system 304. Control system 304 may correspond to or may be part of computing system 172. The tracking error may be a result of a joint (or multiple joints) not being able to reach the commanded position as a result of the contact condition. This may occur, for example, when a link driven by a joint is blocked by another link.

[0084] In one or more embodiments, the joint tracking error(s) is / are processed to determine the resulting end effector tracking error, e.g., at the tip of the end effector of theinstrument supported by the manipulator arm associated with the tracking error. Forward kinematics, e.g., Jacobian-based, may be used to perform this operation.

[0085] In some embodiments proportional (P) controllers or proportional derivative (PD) controllers are used to command joint movement (e.g., a joint position). As a result of the proportional control scheme, the torque or force generated at a joint is directly proportional to the tracking error when the actual joint position fails to reach the commanded joint position. In other words, smaller tracking errors result in smaller forces or torques generated at the joint (to overcome the tracking error) and larger tracking error result in larger forces or torques at the joint. Because, in case of a contact condition, the tracking error is a result of the contact condition, the force or torque associated with the tracking error may be translated to a contact force. Based on the known kinematics of the manipulator arm assembly with the tracking error(s) the actual contact force may be estimated, analogous to how the end effector tracking error is estimated.

[0086] The tracking error or the contact force may, thus, also be used to detect the presence of a contact condition, when the magnitude of the tracking error or contact force exceeds a specified threshold.

[0087] A detailed discussion of the use of the sensor output-based detection of a contact condition and the tracking error-based detection of a contact condition, either separately or in combination, is provided below in reference to FIG. 4.

[0088] In one or more embodiments, the determination of contact conditions can be based further on kinematics information of the manipulator arm assemblies. The kinematics information used in the determination of contact conditions can be determined or estimated by a kinematics engine 302 and can include, for example, a determined or estimated distance between two manipulator arm assemblies, where the kinematics engine 302 is configured to estimate a closest distance between the manipulator arm assemblies.

[0089] In some embodiments, the determination of a contact condition based on detection of impact signatures and / or based on detection of tracking errors or contact forces can be gated by a distance estimate provided by a kinematics engine 302 (e.g., selectively enabled or disabled based on the distance estimate).

[0090] If the closest distance between the manipulator arm assemblies is above a specified threshold (i.e., there is sufficient spatial separation between the manipulator arm assemblies), the implication is that the presence of a contact condition is not likely or even possible. In this case, the contact condition determination engine 310 can be prevented from performing the determination of contact conditions in order to avoid possible false positivedetections of contact conditions caused by unrelated events that produce simultaneous or near-simultaneous impact signatures on the sensor outputs of multiple impact sensors. An example of an event that could produce a false positive is a person accidentally bumping into the repositionable assembly thereby causing a simultaneous presence of impact signatures on multiple sensor outputs.

[0091] If the closest distance is below the specified threshold, the implication is that a contact condition is possible, based on the proximity of the manipulator arm assemblies. In this case, the contact determination engine 310 can be enabled to perform the determination of contact conditions based on the presence of impact signatures on the sensor outputs. The specified threshold may be tuned to eliminate false positives while ensuring that the contact condition determination engine is active when the possibility of contact conditions exists.

[0092] In one or more embodiments, the kinematics engine 302 determines the closest distance by monitoring the kinematic configuration of the manipulator arm assemblies over time. For the purpose of describing the operation of the kinematics engine, assume that there are manipulator arm assemblies A, B, and C. The kinematics engine may have access to the instantaneous joint configurations of the joints of manipulator arm assemblies A, B, and C. Based on the instantaneous joint configurations, the kinematics engine may determine the spatial configuration of the links of manipulator arm assemblies A, B, and C. Next, based on the known spatial configuration of the links and their geometries, the distance between the links can be estimated. For example, distances may be determined between links of manipulator arm assemblies A and B, manipulator arm assemblies A and C, and manipulator arm assemblies B and C. The links of the manipulator arm assemblies may be represented in different manners, to perform the determination of the distance. For example, a volume model may be used to represent a link with a complex geometry (e.g., links 204, 206, 208, 210 in FIG. 2A). Alternatively, 3D line segment models may be used to represent simple rotation-symmetric links. An instrument shaft (e.g., instrument shaft 218 in FIG. 2A) may be represented with good accuracy using a 3D line segment, for example. In this example, the line segment intended to represent the instrument shaft may be defined by the remote center establishing a first point in space and the interface between the distal end of the instrument shaft and the end effector establishing a second point in space. Any reference frame, e.g., the world frame, may be used to represent the line segments. 3D line segment models may be preferred when a reduction of the computational effort associated with the distance estimation is desirable, e.g., when a distance estimate is desired in real-time at a high sampling rate.

[0093] Depending on whether the volume model or the 3D line segment model is used, different thresholds may be applied to determine whether to perform the contact condition detection. For example, a 10 mm threshold may be used in conjunction with the volume model, and a 5mm threshold may be used in conjunction with the 3D line segment model.

[0094] When reporting a closest distance to the contact condition determination engine 310, a location of the closest distance (i.e., identifying where the links are in the contact condition) may be provided as well.

[0095] Once the contact condition determination engine 310 has determined that a contact condition is present (either using the sensor output-based detection or the tracking error-based detection), the contact condition determination engine 310 may set a flag to indicate that the contact condition is present. The flag may be provided to the user interface (UI) 320. In addition, the contact force may also be provided to the UI 320. The UI 320 may, thus, notify the user(s) of the computer-assisted system of the contact condition. The notification may include a message or status indicator displayed in one more displays, an auditory signal, a tactile feedback, etc. In one or more embodiments, the contact force is used to gate the providing of a notification to the user(s). For example, the UI may provide a notification only if the contact force exceeds a specified threshold, e.g., 0.5N or any other specified contact force threshold. Various aspects of user interfaces are described in U.S. Provisional Patent Application No. 63 / 724,173, titled “SYSTEMS AND METHODS FOR INDICATING AN EVENT OUTSIDE OF AN IMAGING SYSTEM FIELD OF VIEW”, which is incorporated herein by reference.

[0096] FIG. 4 shows a flowchart describing example methods for detecting contact conditions in accordance with one or more embodiments. The method may be used to detect contact conditions between manipulator arm assemblies of computer-assisted systems. One or more of the steps in FIG. 4 may be performed by various components of systems, previously described with reference to FIGs. 1, 2 A, 2B, and 3. While these figures illustrate particular configurations of computer assisted systems, the method is equally applicable to other configurations. The method may be executed on one or more processors, e.g., of the control system of the computer-assisted system.

[0097] While the various steps in the flowchart are presented and described sequentially, one of ordinary skill will appreciate that some or all of the steps may be executed in different orders, may be combined or omitted, and some or all of the steps may be executed in parallel. Additional steps may further be performed. Furthermore, the stepsmay be performed actively or passively. For example, some steps may be performed using polling or be interrupt driven in accordance with one or more embodiments of the invention.

[0098] Assume that the method is executed for a computer-assisted system that comprises at least two manipulator arm assemblies. For example, assume that the computer-assisted system comprises a manipulator arm assembly A with a manipulator arm A that is configured to support an instrument A, and that the computer-assisted system further comprises a manipulator arm assembly B with a manipulator arm B that is configured to support an instrument B. Additional manipulator arm assemblies such as manipulator arm assemblies C and D may be present.

[0099] As previously discussed in reference to FIG. 3, in some embodiments the execution of the method relies on a sensor output A from an impact sensor A (disposed on manipulator arm A) and a sensor output B from an impact sensor B (disposed on manipulator arm B). The method may further rely on a distance estimate of a closest distance between the manipulator arm assembly A and manipulator arm assembly B. The distance estimate may be provided by a kinematics engine. The sensor outputs A and B may be obtained at any rate, e.g., a fixed sampling rate. Similarly, the distance estimate may be obtained at any rate.

[0100] Further, as previously discussed in FIG. 3, in some embodiments the execution of the method relies on a tracking error obtained from the control system of the computer-assisted system. The method may further rely on a distance estimate of a closest distance between the manipulator arm assembly A and manipulator arm assembly B. The distance estimate may be provided by a kinematics engine. The tracking error may be obtained at any rate, e.g., a fixed sampling rate. Similarly, the distance estimate may be obtained at any rate.

[0101] The method may be executed for one or both of the sensor output-based detection and the contact force-based detection.

[0102] Turning to the flowchart, in Step 402 a determination is made whether the distance estimate is below a specified threshold. If the distance estimate is below the specified threshold the method may proceed with the execution of step 404. If the distance estimate is not below the specified threshold, it may be determined that there is no contact condition (absence of a contact condition) without execution of at least some of steps 404-412. As previously discussed, the determining of the distance estimate may be performed across multiple or all manipulator arm assemblies to ensure that any possible combination of links that are in close proximity (based on the distance estimate) can be detected. Based on the conditional execution of the subsequent steps, a contact condition may then be detectedfor the pair(s) of manipulator arm assemblies that was / were found to be in close proximity, as a result of the execution of Step 402.

[0103] In Step 404, a determination is made whether a contact condition is present. As previously noted, the determination may be made in different manners.

[0104] In some embodiments, the determination is made based on sensor outputs A and B as described in reference to FIG. 3. If impact signatures are present on sensor outputs A and B within a detection time window, the execution of the method may proceed to Step 406, where it is determined that the contact condition is present. If an impact signature is not detected on both sensor outputs A and B within the detection time window, the execution of the method may return to Step 402.

[0105] In some embodiments, the determination is made based on a tracking error or corresponding estimated contact force, as described in reference to FIG. 3. If the tracking error exceeds a specified threshold, the method may proceed to Step 406, where it is determined that the contact condition is present. If the tracking error does not exceed the specified threshold, the execution of the method may return to step 402.

[0106] In Step 408, one or more user interface elements relating to the contact condition are presented via a user interface to report the contact condition to the user. In addition to the presence of the contact condition, the location of the contact condition may be reported using a user interface element. For example, the location may be identified in space, the contact condition may be identified as external or internal to the barrier, or the links involved in the contact condition may be identified. For a collision internal to the barrier, the location of the contact condition may be identified in the view provided to the user, if the contact condition is located in the field of view. If the contact condition is outside the field of view, direction indicators (e.g., arrows) may be provided to indicate the location of the contact condition relative to the field of view, or to guide the user to change the field of view to obtain a view of the contact condition. Whether or not the contact condition is in the field of view may be determined by considering the location of the contact condition relative to the field of view which is known based on the spatial configuration of the imaging device and its optical configuration which may define the field of view.

[0107] The location of the contact condition may be determined based on the location of the closest distance obtained from the kinematics engine. Additional information, beyond the location of the contact condition may further be provided. For example, the components of the manipulator am assemblies involved in the contact condition may be identified. At the highest level, the manipulator arm assemblies may be identified, e.g., by a user interfaceelement identifying manipulator arm assemblies A and B as being involved in the contact condition. The information needed to provide this information may be obtained based on the execution of Steps 404 and 406 which led to the detection of a contact condition (in the example, a contact condition for the combination of manipulator arm assemblies A and B, but not for other pairs of manipulator arm assemblies). To provide the user with additional insights regarding the contact condition, the links involved in the contact condition may be reported. These links may be the links for which the closest distance was determined during the execution of Step 402. Based on the links found to be involved in the contact condition, the contact condition may be identified as an arm-to-arm or instrument shaft-to-instrument shaft contact condition, using a user interface element.

[0108] In some embodiments, a contact type (e.g., manipulator arm-to-manipulator arm vs. instrument shaft-to-instrument shaft) of the contact condition is determined, at least in part, based on detectable differences in the corresponding IMU signatures. For example, the frequency components found in the IMU signatures may differ between a contact between manipulator arms as compared to a contact between instrument shafts.

[0109] Further, a classification of the contact condition may be provided in a user interface element. For example, the contact condition may be classified as a hard collision based on having been detected using the sensor output-based method, or as a soft collision, based on having been detected using the tracking error-based method, but not the sensor output-based method. The contact condition may also be classified as a contact condition internal to the barrier or external to the barrier, as a contact condition between manipulator arms or instruments, etc. The contact force may be provided in an additional user interface element. In some embodiments, possible consequences of the contact condition (e.g., limited range of motion of the instrument(s)) may be identified in a user interface element and / or suggestions for the resolution of the contact condition may be provided, e.g., by guiding the user to perform a movement that reduces the contact force. The reporting as described may rely on a user interface that may include visual, auditory and / or haptic aspects. In one or more embodiments, the visual and haptic feedback may be provided at the onset of the contact condition, whereas the visual feedback may be provided until the contact condition is resolved.

[0110] As previously discussed, whether the user interface element that relates to the contact condition is presented may depend on whether the contact force exceeds associated with the contact condition exceeds a specified threshold. Accordingly, if the contact forcedoes not exceed the specified threshold, the user interface element may not be displayed, or only certain aspects of it may be displayed.

[0111] In Step 410, a determination is made whether the distance estimate is below the specified threshold. If the distance estimate is still below the specified threshold, it is concluded that the contact condition remains present. If the distance estimate is above the specified threshold the method may proceed with the execution of Step 412, where it is determined that the contact condition is no longer present.

[0112] Briefly summarized, the execution of Steps 402 and 410, thus, gates the detection of a contact condition using an estimated distance. The detection of the contact condition is performed only if the estimated distance drops below a specified threshold. Once a contact condition has been detected, the contact condition is detected as present until a newly obtained distance estimate is above the specified threshold. An example of the execution of the method of FIG. 4 based on sensor outputs obtained from IMUs is shown in FIG. 5.

[0113] FIG. 5 shows a set of plots that illustrate the execution of the method of FIG.4. The upper plot shows sensor outputs A and B over time. In the example, sensor outputs A and B are acceleration signals. In the center plot the distance estimate of the closest distance between manipulator arm assembly A and manipulator arm assembly B is shown over time. The specified threshold distance is identified as well. In the bottom plot, a flag that indicates whether a contact condition has been detected is shown. The set of plots in combination illustrates a dip of the distance estimate to below the specified threshold, followed by impact signatures on both sensor outputs A and B, within a detection time window. Because the detection time window is present at a time when the distance estimate is below the specified threshold, the contact condition detected flag is activated. While no additional impact signatures are visible while the distance estimate is below the specified threshold, the contact detection flag is kept active, until the distance estimate increase to exceed the specified threshold. This is when the contact condition detected flag is disabled.

[0114] In some embodiments, the sensor outputs of the impact sensors may be used to determine a gravity vector, e.g., using an impact sensor that includes an IMU. A gravitybased determination of contact conditions can thus be implemented. The gravity -based determination of a contact condition may be based on an interpretation of the static or nearstatic effect of gravity on an IMU by observing the gravity vector as it acts on the IMU over time. Consider a scenario of a stationary IMU (e.g., disposed on a first manipulator arm assembly that is not moving). In this scenario, gravity is expected to act on the IMU in aknown and continuous manner. When a second manipulator arm assembly comes in contact with the first manipulator arm assembly, this may result in a small change in the orientation of the IMU disposed on the first manipulator arm assembly. This orientational change may be detectable based on the change of how gravity acts on the IMU, thereby possibly serving as an indication of a contact condition. Generalizing to scenarios that do not impose the constraint of the first manipulator arm assembly being stationary, by compensating for acceleration effects associated with movement, the gravity-based detection method may also be performed in presence of movement of the first manipulator arm assembly. The method may further be extended to reject common disturbances such as respiratory movement. The gravity -based detection may be incorporated in the method previously described in reference to FIG. 4, e.g., by replacing or supplementing the determination made in Step 406 with the gravity-based determination.

[0115] Changes in how the gravity vector acts on an IMU may further be used for other purposes. For example, a remote center that is poorly aligned with an actual incision (or, more generally, an orifice) may result in persistent transmission of body wall forces and / or torques from the body wall to the manipulator arms. These forces / torques may be detected as shifts in the gravity vector registered by the IMU (example: respiratory movement in which the shift is periodic). A warning may be issued to the user, based on this detection. Readjustment of the remote center to better align with the orifice may then reduce or eliminate the forces / torques on the manipulator arm.

[0116] Although a limited number of example embodiments have been described in detail above, those skilled in the art will readily appreciate that many modifications are possible in the example embodiments without materially departing from this disclosure. Accordingly, all such modifications are intended to be included within the scope of this disclosure as defined in the following claims.

Claims

CLAIMSWhat is claimed is:

1. A computer-assisted system comprising:a first manipulator arm assembly comprising a first manipulator arm, the first manipulator arm configured to support a first instrument;a second manipulator arm assembly comprising a second manipulator arm, the second manipulator arm configured to support a second instrument;a first impact sensor disposed on the first manipulator arm assembly;a second impact sensor disposed on the second manipulator arm assembly; and a control system configured to:detect, within a detection time window, a first impact signature based on a first sensor output received from the first impact sensor;detect, within the detection time window, a second impact signature based on a second sensor output received from the second impact sensor; and determine that a contact condition is present between the first manipulator arm assembly and the second manipulator arm assembly, based on the detection of the first impact signature and the detection of the second impact signature within the detection time window.

2. The computer-assisted system of claim 1, further comprising:a third manipulator arm assembly comprising a third manipulator arm, the third manipulator arm configured to support a third instrument;a third impact sensor disposed on the third manipulator arm,wherein the control system is further configured to determine whether the contact condition involves the third manipulator arm assembly by:obtaining a third sensor output from the second impact sensor;detecting, within the detection time window, an absence of a third impact signature on the third sensor output; anddetermining that a contact condition involving the third manipulator arm assembly is absent, based on the absence of the third impact signature.

3. The computer-assisted system of claim 1,wherein the control system is further configured to obtain a first distance estimate of a closest distance between the first manipulator arm assembly and the second manipulator arm assembly, andwherein the determination that the contact condition between the first manipulator arm assembly and the second manipulator arm assembly is present is based on the first distance estimate being below a specified threshold.

4. The computer-assisted system of claim 3,wherein the determination that the contact condition between the first manipulator arm assembly and the second manipulator arm assembly is present requires the first distance estimate to be below the specified threshold.

5. The computer-assisted system of claim 3, wherein the control system is further configured to:obtain a second distance estimate of the closest distance between the first manipulator arm assembly and the second manipulator arm assembly after determining that the contact condition is present; anddetermine that the contact condition remains present based on the second distance estimate being below the specified threshold.

6. The computer-assisted system of claim 5, further comprising:a third manipulator arm assembly comprising a third manipulator arm, the third manipulator arm configured to support a third instrument;wherein the control system is further configured to:obtain a second distance estimate of a closest distance between the first manipulator arm assembly and the third manipulator arm assembly; andbased on the second distance estimate being above the specified threshold, determining an absence of a contact condition between the first manipulator arm assembly and the third manipulator arm assembly.

7. The computer-assisted system of claim 3,wherein the control system is further configured to obtain a second distance estimate of the closest distance between the first manipulator arm assembly and thesecond manipulator arm assembly after determining that the contact condition is present, andwherein the determining that the contact condition is no longer present is based on the second distance estimate being above the specified threshold.

8. The computer-assisted system of claim 3,wherein to obtain the first distance estimate, the control system is configured to represent an element of the first manipulator arm assembly using a volume model, andwherein the control system is configured to obtain the first distance estimate using the volume model.

9. The computer-assisted system of claim 3,wherein to obtain the first distance estimate, the control system is configured to represent an element of the first manipulator arm assembly using a line segment model, andwherein the control system is configured to obtain the first distance estimate using the line segment model.

10. The computer-assisted system of claim 1, wherein the control system is further configured to determine a contact type, at least in part based, on at least one of the first sensor output and the second sensor output.

11. The computer-assisted system of any of claims 1 to 10, wherein the control system is further configured to present a user interface element relating to the contact condition via a user interface.

12. The computer-assisted system of claim 11, wherein the user interface element identifies the first and second manipulator arm assemblies as being associated with the contact condition.

13. The computer-assisted system of claim 11, wherein the user interface element identifies that the contact condition is an arm-to-arm or an instrument shaft-to-instrument shaft contact condition.

14. The computer-assisted system of claim 11,wherein the control system is further configured to estimate a contact force associated with the contact condition, andwherein the presenting of the user interface element relating to the contact condition via the user interface is based on determining that the contact force exceeds a specified threshold.

15. The computer-assisted system of claim 11, wherein to present the user interface element relating to the contact condition via the user interface, the control system is configured to identify a location of the contact condition.

16. The computer-assisted system of claim 15, wherein to identify the location of the contact condition, the control system is configured to use a kinematic model of the first manipulator arm assembly and of the second manipulator arm assembly.

17. The computer-assisted system of claim 15, wherein to identify the location of the contact condition, the control system is configured to identify the location in a view obtained from an imaging instrument.

18. The computer-assisted system of claim 17, wherein to identify the location of the contact condition, the control system is further configured to identify the location of the contact condition in an image of the first instrument in the view.

19. The computer-assisted system of claim 17, wherein to identify the location of the contact condition, the control system is further configured to identify the location of the contact condition using a direction indicator when the location is outside the view.

20. The computer-assisted system of any of claims 1 to 10, further comprising:an input device configured to receive a user input for teleoperating the first instrument, andwherein the control system is further configured to:control the input device to provide a first haptic feedback that reflects the contact condition.

21. The computer-assisted system of claim 20, wherein the control system is further configured to:control the input device to provide a second haptic feedback that reflects a force at an end effector of the first instrument, andprovide an indication that the first haptic feedback is associated with the contact condition, based on the determination that the contact condition is present.

22. The computer-assisted system of any of claims 1 to 10, wherein the first impact sensor is an inertial measurement unit (IMU).

23. A computer-assisted system comprising:a first manipulator arm assembly comprising a first manipulator arm, the first manipulator arm configured to support a first instrument;a second manipulator arm assembly comprising a second manipulator arm, the second manipulator arm configured to support a second instrument; and a control system configured to:determine a contact force between the first manipulator arm assembly and the second manipulator arm assembly based on a tracking error in at least one of a plurality of joints of the first and the second manipulator arm; anddetermine that a contact condition is present between the first manipulator arm assembly and the second manipulator arm assembly, based on the contact force exceeding a pre-specified threshold.

24. The computer-assisted system of claim 23,wherein the control system is further configured to obtain a first distance estimate of a closest distance between the first manipulator arm assembly and the second manipulator arm assembly, andwherein the determination that the contact condition between the first manipulator arm assembly and the second manipulator arm assembly is present is based on the first distance estimate being below a specified threshold.

25. The computer-assisted system of claim 24,wherein the determination that the contact condition between the first manipulator arm assembly and the second manipulator arm assembly is present requires the first distance estimate to be below the specified threshold.

26. The computer-assisted system of claim 24, wherein the control system is further configured to:obtain a second distance estimate of the closest distance between the first manipulator arm assembly and the second manipulator arm assembly after determining that the contact condition is present; anddetermine that the contact condition remains present based on the second distance estimate being below the specified threshold.

27. The computer-assisted system of claim 26, further comprising:a third manipulator arm assembly comprising a third manipulator arm, the third manipulator arm configured to support a third instrument;wherein the control system is further configured to:obtain a second distance estimate of a closest distance between the first manipulator arm assembly and the third manipulator arm assembly; andbased on the second distance estimate being above the specified threshold, determining an absence of a contact condition between the first manipulator arm assembly and the third manipulator arm assembly.

28. The computer-assisted system of claim 24,wherein the control system is further configured to obtain a second distance estimate of the closest distance between the first manipulator arm assembly and the second manipulator arm assembly after determining that the contact condition is present, andwherein the determining that the contact condition is no longer present is based on the second distance estimate being above the specified threshold.

29. The computer-assisted system of claim 24,wherein to obtain the first distance estimate, the control system is configured to represent an element of the first manipulator arm assembly using a volume model, andwherein the control system is configured to obtain the first distance estimate using the volume model.

30. The computer-assisted system of claim 24,wherein to obtain the first distance estimate, the control system is configured to represent an element of the first manipulator arm assembly using a line segment model, andwherein the control system is configured to obtain the first distance estimate using the line segment model.

31. The computer-assisted system of any of claims 23 to 30, wherein the control system is further configured to present a user interface element relating to the contact condition via a user interface.

32. The computer-assisted system of claim 31, wherein the user interface element identifies the first and second manipulator arm assemblies as being associated with the contact condition.

33. The computer-assisted system of claim 31, wherein the user interface element identifies that the contact condition is an arm-to-arm or an instrument shaft-to-instrument shaft contact condition.

34. The computer-assisted system of claim 31,wherein the control system is further configured to estimate a contact force associated with the contact condition, andwherein the presenting of the user interface element relating to the contact condition via the user interface is based on determining that the contact force exceeds a specified threshold.

35. The computer-assisted system of claim 31, wherein to present the user interface element relating to the contact condition via the user interface, the control system is configured to identify a location of the contact condition.

36. The computer-assisted system of claim 35, wherein to identify the location of the contact condition, the control system is configured to use a kinematic model of the first manipulator arm assembly and of the second manipulator arm assembly.

37. The computer-assisted system of claim 35, wherein to identify the location of the contact condition, the control system is configured to identify the location in a view obtained from an imaging instrument.

38. The computer-assisted system of claim 37, wherein to identify the location of the contact condition, the control system is further configured to identify the location of the contact condition in an image of the first instrument in the view.

39. The computer-assisted system of claim 37, wherein to identify the location of the contact condition, the control system is further configured to identify the location of the contact condition using a direction indicator when the location is outside the view.

40. The computer-assisted system of any of claims 23 to 30, further comprising:an input device configured to receive a user input for teleoperating the first instrument, andwherein the control system is further configured to:control the input device to provide a first haptic feedback that reflects the contact condition.

41. The computer-assisted system of claim 40, wherein the control system is further configured to:control the input device to provide a second haptic feedback that reflects a force at an end effector of the first instrument, andprovide an indication that the first haptic feedback is associated with the contact condition, based on the determination that the contact condition is present.

42. A method of operating a computer-assisted system,the computer-assisted system comprising:a first manipulator arm assembly comprising a first manipulator arm, the first manipulator arm configured to support a first instrument;a second manipulator arm assembly comprising a second manipulator arm, the second manipulator arm configured to support a second instrument; a first impact sensor disposed on the first manipulator arm assembly;a second impact sensor disposed on the second manipulator arm assembly; and the method comprising:detecting, within a detection time window, a first impact signature based on a first sensor output received from the first impact sensor; detecting, within the detection time window, a second impact signature based on a second sensor output received from the second impact sensor; and determining that a contact condition is present between the first manipulator arm assembly and the second manipulator arm assembly, based on the detection of the first impact signature and the detection of the second impact signature within the detection time window.

43. The method of claim 42,wherein the computer-assisted system further comprises:a third manipulator arm assembly comprising a third manipulator arm, the third manipulator arm configured to support a third instrument;a third impact sensor disposed on the third manipulator arm; and wherein the method further comprises determining whether the contact condition involves the third manipulator arm assembly by:obtaining a third sensor output from the second impact sensor; detecting, within the detection time window, an absence of a third impact signature on the third sensor output; anddetermining that a contact condition involving the third manipulator arm assembly is absent, based on the absence of the third impact signature.

44. The method of claim 43, further comprising:obtaining a first distance estimate of a closest distance between the first manipulator arm assembly and the second manipulator arm assembly, andwherein the determination that the contact condition between the first manipulator arm assembly and the second manipulator arm assembly is present is based on the first distance estimate being below a specified threshold.

45. The method of claim 44,wherein the determination that the contact condition between the first manipulator arm assembly and the second manipulator arm assembly is present requires the first distance estimate to be below the specified threshold.

46. The method of claim 44, further comprising:obtaining a second distance estimate of the closest distance between the first manipulator arm assembly and the second manipulator arm assembly after determining that the contact condition is present; anddetermining that the contact condition remains present based on the second distance estimate being below the specified threshold.

47. The method of claim 46,wherein the computer-assisted system further comprises:a third manipulator arm assembly comprising a third manipulator arm, the third manipulator arm configured to support a third instrument; wherein the method further comprises:obtaining a second distance estimate of a closest distance between the first manipulator arm assembly and the third manipulator arm assembly; andbased on the second distance estimate being above the specified threshold, determining an absence of a contact condition between the first manipulator arm assembly and the third manipulator arm assembly.

48. The method of claim 44, further comprising:obtaining a second distance estimate of the closest distance between the first manipulator arm assembly and the second manipulator arm assembly after determining that the contact condition is present, andwherein the determining that the contact condition is no longer present is based on the second distance estimate being above the specified threshold.

49. The method of claim 44,wherein the obtaining of the first distance estimate comprises representing an element of the first manipulator arm assembly using a volume model, and wherein the method further comprises obtaining the first distance estimate using the volume model.

50. The method of claim 44,wherein the obtaining of the first distance estimate comprises representing an element of the first manipulator arm assembly using a line segment model, and wherein the method further comprises obtaining the first distance estimate using the line segment model.

51. The method of claim 42, further comprising determining a contact type, at least in part based, on at least one of the first sensor output and the second sensor output.

52. The method of any of claims 43 to 51, further comprising presenting a user interface element relating to the contact condition via a user interface.

53. The method of claim 52, wherein the user interface element identifies the first and second manipulator arm assemblies as being associated with the contact condition.

54. The method of claim 52, wherein the user interface element identifies that the contact condition is an arm-to-arm or an instrument shaft-to-instrument shaft contact condition.

55. The method of claim 52, further comprising:estimating a contact force associated with the contact condition, andwherein the presenting of the user interface element relating to the contact condition via the user interface is based on determining that the contact force exceeds a specified threshold.

56. The method of claim 52, wherein the presenting of the user interface element relating to the contact condition via the user interface comprises identifying a location of the contact condition.

57. The method of claim 56, wherein the identifying of the location of the contact condition comprises using a kinematic model of the first manipulator arm assembly and of the second manipulator arm assembly.

58. The method of claim 56, wherein the identifying of the location of the contact condition comprises identifying the location in a view obtained from an imaging instrument.

59. The method of claim 58, wherein the identifying of the location of the contact condition further comprises identifying the location of the contact condition in an image of the first instrument in the view.

60. The method of claim 58, wherein the identifying of the location of the contact condition further comprises identifying the location of the contact condition using a direction indicator when the location is outside the view.

61. The method of any of claims 43 to 51,wherein the computer-assisted system further comprises an input device configured to receive a user input for teleoperating the first instrument; andwherein the method further comprises controlling the input device to provide a first haptic feedback that reflects the contact condition.

62. The method of claim 61, further comprising:controlling the input device to provide a second haptic feedback that reflects a force at an end effector of the first instrument; andproviding an indication that the first haptic feedback is associated with the contact condition, based on the determination that the contact condition is present.

63. The method of any of claims 43 to 51, wherein the first impact sensor is an inertial measurement unit (IMU).

64. A non-transitory machine-readable medium comprising a plurality of machine-readable instructions executed by one or more processors associated with a computer-assisted system, the plurality of machine-readable instructions causing the one or more processors to perform the method of any of claims 42 to 63.

65. A method of operating a computer-assisted system,the computer-assisted system comprising:a first manipulator arm assembly comprising a first manipulator arm, the first manipulator arm configured to support a first instrument;a second manipulator arm assembly comprising a second manipulator arm, the second manipulator arm configured to support a second instrument; andthe method comprising:determining a contact force between the first manipulator arm assembly and the second manipulator arm assembly based on a tracking error in at least one of a plurality of joints of the first and the second manipulator arm; anddetermining that a contact condition is present between the first manipulator arm assembly and the second manipulator arm assembly, based on the contact force exceeding a pre-specified threshold.

66. The method of claim 65, further comprising:obtaining a first distance estimate of a closest distance between the first manipulator arm assembly and the second manipulator arm assembly, andwherein the determination that the contact condition between the first manipulator arm assembly and the second manipulator arm assembly is present is based on the first distance estimate being below a specified threshold.

67. The method of claim 66,wherein the determination that the contact condition between the first manipulator arm assembly and the second manipulator arm assembly is present requires the first distance estimate to be below the specified threshold.

68. The method of claim 66, further comprising:obtaining a second distance estimate of the closest distance between the first manipulator arm assembly and the second manipulator arm assembly after determining that the contact condition is present; anddetermining that the contact condition remains present based on the second distance estimate being below the specified threshold.

69. The method of claim 68,wherein the computer-assisted system further comprises:a third manipulator arm assembly comprising a third manipulator arm, the third manipulator arm configured to support a third instrument; wherein the method further comprises:obtaining a second distance estimate of a closest distance between the first manipulator arm assembly and the third manipulator arm assembly; andbased on the second distance estimate being above the specified threshold, determining an absence of a contact condition between the first manipulator arm assembly and the third manipulator arm assembly.

70. The method of claim 66, further comprising:obtaining a second distance estimate of the closest distance between the first manipulator arm assembly and the second manipulator arm assembly after determining that the contact condition is present, andwherein the determining that the contact condition is no longer present is based on the second distance estimate being above the specified threshold.

71. The method of claim 66,wherein the obtaining of the first distance estimate comprises representing an element of the first manipulator arm assembly using a volume model, and wherein the method further comprises obtaining the first distance estimate using the volume model.

72. The method of claim 66,wherein the obtaining of the first distance estimate comprises representing an element of the first manipulator arm assembly using a line segment model, and wherein the method further comprises obtaining the first distance estimate using the line segment model.

73. The method of any of claims 64 to 72, further comprising presenting a user interface element relating to the contact condition via a user interface.

74. The method of claim 73, wherein the user interface element identifies the first and second manipulator arm assemblies as being associated with the contact condition.

75. The method of claim 73, wherein the user interface element identifies that the contact condition is an arm-to-arm or an instrument shaft-to-instrument shaft contact condition.

76. The method of claim 73, further comprising:estimating a contact force associated with the contact condition, andwherein the presenting of the user interface element relating to the contact condition via the user interface is based on determining that the contact force exceeds a specified threshold.

77. The method of claim 73, wherein the presenting of the user interface element relating to the contact condition via the user interface comprises identifying a location of the contact condition.

78. The method of claim 77, wherein the identifying of the location of the contact condition comprises using a kinematic model of the first manipulator arm assembly and of the second manipulator arm assembly.

79. The method of claim 77, wherein the identifying of the location of the contact condition comprises identifying the location in a view obtained from an imaging instrument.

80. The method of claim 79, wherein the identifying of the location of the contact condition further comprises identifying the location of the contact condition in an image of the first instrument in the view.

81. The method of claim 79, wherein the identifying of the location of the contact condition further comprises identifying the location of the contact condition using a direction indicator when the location is outside the view.

82. The method of any of claims 64 to 72,wherein the computer-assisted system further comprises an input device configured to receive a user input for teleoperating the first instrument, andwherein the method further comprises controlling the input device to provide a first haptic feedback that reflects the contact condition.

83. The method of claim 82, further comprising:controlling the input device to provide a second haptic feedback that reflects a force at an end effector of the first instrument, andproviding an indication that the first haptic feedback is associated with the contact condition, based on the determination that the contact condition is present.

84. A non-transitory machine-readable medium comprising a plurality of machine-readable instructions executed by one or more processors associated with a computer-assisted system, the plurality of machine-readable instructions causing the one or more processors to perform the method of any of claims 65 to 83.

Images