Repositioning an instrument supported by a repositionable assembly of a computer-assisted system with an input system

Abstract

A computer-assisted system includes a user input system and a repositionable assembly. The user input system is configured to receive an input from a user and the repositionable assembly is configured to support one or more instruments. The one or more instruments include an imaging device. The computer-assisted system further includes a control system with one or more processors and communicatively coupled to the user input system and the repositionable assembly. The control system is configured to determine a reference frame being offset from an instrument frame of the imaging device. The control system is further configured to receive, at the user input system, a user input to reposition or reorient the imaging device and determine, based on the user input, a change in the reference frame. The control system is further configured to provide, based on the change in the reference frame, feedback to the user input system.

Description

REPOSITIONING AN INSTRUMENT SUPPORTED BY A REPOSITIONABLE ASSEMBLY OF A COMPUTER-ASSISTED SYSTEM WITH AN INPUT SYSTEM CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to and the benefit of the filing date of U.S. Provisional Patent Application 63 / 753,584, filed on February 4, 2025, which is hereby incorporated by reference herein in its entirety.BACKGROUNDField of Invention

[0002] The present invention generally provides improved computer-assisted devices, systems, and methods.Overview

[0003] Computer-assisted systems can be used to perform a task at a worksite.Example computer-assisted systems include industrial and recreational robotic systems. Example computer-assisted systems also include medical robotic systems used in procedures for diagnosis, non-surgical treatment, surgical treatment, etc. The computer-assisted system may include a repositionable assembly configured to support instruments. An instrument may be an instrument for performing the task. An instrument may also be an instrument for viewing the performing of the task. The computer-assisted system may be equipped with any number of instruments of any type.

[0004] The computer-assisted system can include a user input system including one or more input devices. As an example, in a computer-assisted system configured for teleoperation, an operator at a user input system manipulates a leader device (e.g., an input device configured to accept commands for a follower device) to cause motions of a follower device (e.g., a repositionable assembly that can be teleoperated).

[0005] Operation of the computer-assisted system can include the reception of a user input at a user input system including an input device to reconfigure the repositionable assembly, e.g., to reposition a supported instrument. Further, feedback can be provided at the input device. The feedback can include the restriction of motion of the input device based on a constraint in the reconfiguration of the repositionable assembly. For example, a degree of freedom of the repositionable assembly can reach a range of motion limit preventing further reconfiguration of the repositionable assembly with respect to, at least, that degree offreedom. As another example, the repositionable assembly may be constrained to maintain, with the degrees of freedom of an instrument, the global position of an instrument. That is, degrees of freedom of the instrument can be used to maintain the global position of the instrument in response to the reconfiguration of the repositionable assembly that supports the instrument, and the reconfiguration of the repositionable assembly can be constrained to configurations that do not violate the range of motion limits of the degrees of freedom of the instrument. In review, feedback can be provided at the input device where the feedback is indicative of a range of motion limit of one or more of the repositionable assembly and a supported instrument.

[0006] In situations where an operator would like to reposition a supported instrument, motions of the input device relating to an operator frame of reference (“operator frame”) can be used to determine corresponding motion commands for the repositionable assembly relating to an instrument frame of reference (“instrument frame”) of the supported instrument. For example, a mapping, such as a rigid and scaled connection, between the operator frame of reference and the instrument frame of reference can be established.However, in some situations, the instrument frame cannot move according to a movement of the operator frame. That is, the repositionable assembly cannot be reconfigured in such a manner as to move the instrument frame according to movements of the operator frame. Examples of such situations include instances where the degrees of freedom of the input device and the repositionable assembly are mismatched either in number of degrees of freedom or type of degrees of freedom. Thus, in these situations, feedback may be rendered to a user through the input device originating from kinematic differences between the input device and the repositionable assembly with respect to their frames of reference rather than, for example, a constraint based on a range of motion limit of the instrument or repositionable assembly.

[0007] Operation of computer-assisted systems may be improved by mitigating feedback provided at an input device originating from a kinematic inability of an instrument frame to move according to movements of an operator frame.SUMMARY

[0008] In general, in one aspect, one or more embodiments relate to a computer-assisted system including a user input system and a repositionable assembly. The user input system is configured to receive an input from a user and the repositionable assembly is configured to support one or more instruments. The one or more instruments include an imaging device. The computer-assisted system further includes a control system with one ormore processors and communicatively coupled to the user input system and the repositionable assembly. The control system is configured to determine a reference frame being offset from an instrument frame of the imaging device. The control system is further configured to receive, at the user input system, a user input to reposition or reorient the imaging device and determine, based on the user input, a change in the reference frame. The control system is further configured to provide, based on the change in the reference frame, feedback to the user input system.

[0009] In general, in one aspect, one or more embodiments relate to a method for controlling a computer-assisted system, the method performed by a control system of the computer-assisted system. The computer-assisted system includes a repositionable assembly configured to support one or more instruments including an imaging device. The computer-assisted system further includes a user input system. The method includes receiving an user input from the user input system, where the user input is indicative of a command to reposition or reorient the imaging device. The method further includes determining a reference frame, where the reference frame is offset from an instrument frame of the imaging device. The method further includes determining, based on the user input, a change in the reference frame and providing, based on the change in the reference frame, feedback to the user input system.

[0010] In general, in one aspect, one or more embodiments relate to a non-transitory machine-readable medium comprising a plurality of machine-readable instructions executed by one or more processors associated with a computer-assisted system. The plurality of machine-readable instructions causing the one or more processors to perform a method. The computer-assisted system includes a repositionable assembly configured to support one or more instruments including an imaging device. The computer-assisted system further includes an user input system. The method includes receiving an user input from the user input system, where the user input is indicative of a command to reposition or reorient the imaging device. The method further includes determining a reference frame, where the reference frame is offset from an instrument frame of the imaging device. The method further includes determining, based on the user input, a change in the reference frame and providing, based on the change in the reference frame, feedback to the user input system.

[0011] Other aspects of the invention will be apparent from the following description and the appended claims.BRIEF DESCRIPTION OF DRAWINGS

[0012] The accompanying drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component that is illustrated in various figures may be represented by a like numeral. For purposes of clarity, not every component may be labeled in every drawing. In the drawings:

[0013] FIG. 1 shows an example repositionable assembly in accordance with one or more embodiments.

[0014] FIG. 2A shows an example computer-assisted system in accordance with one or more embodiments.

[0015] FIG. 2B shows an example system in accordance with one or more embodiments.

[0016] FIG. 3 shows an example repositionable assembly in accordance with one or more embodiments.

[0017] FIGs. 4A and 4B show example instruments in accordance with one or more embodiments.

[0018] FIG. 5A depicts a portion of a repositionable assembly with supported instruments and an instrument frame, in accordance with one or more embodiments.

[0019] FIGs. 5B to 5C depict examples of an operator frame in accordance with one or more embodiments.

[0020] FIG. 6A depicts a portion of a repositionable assembly with supported instruments and an instrument frame, in accordance with one or more embodiments.

[0021] FIGs. 6B and 6C depict a desired user input and a possible approximate user input, respectively, using two virtually linked input devices in accordance with one or more embodiments.

[0022] FIGs. 7A to 7D depict examples of reconfiguring a repositionable assembly to reposition and / or reorient a supported imaging device using a reference frame in accordance with one or more embodiments.

[0023] FIGs. 8A to 8C depict examples of reconfiguring a repositionable assembly to reposition and / or reorient a supported imaging device using a reference frame in accordance with one or more embodiments.

[0024] FIG. 9 shows a flowchart describing an example method for reconfiguring a repositionable assembly to reposition and / or reorient a supported imaging device using a reference frame in accordance with one or more embodiments.DETAILED DESCRIPTION

[0025] Specific embodiments of the disclosure will now be described in detail with reference to the accompanying figures. Like elements in the various figures are denoted by like reference numerals for consistency.

[0026] In the following detailed description of embodiments of the disclosure, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to one of ordinary skill in the art that the invention may be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the description.

[0027] Throughout the application, ordinal numbers (e.g., first, second, third, etc.) may be used as an adjective for an element (i.e., any noun in the application). The use of ordinal numbers is not to imply or create any particular ordering of the elements, and is not to limit any element to being only a single element unless expressly disclosed, such as by the use of the terms “before”, “after”, “single”, and other such terminology. Rather, the use of ordinal numbers is to distinguish between the elements. By way of an example, a first element is distinct from a second element, and the first element may encompass more than one element and succeed (or precede) the second element in an ordering of elements.

[0028] This disclosure describes various devices, elements, and portions of computer-assisted systems and elements in terms of their state in three-dimensional space. As used herein, the term “position” refers to the location of an element or a portion of an element (c.g, three degrees of translational freedom in a three-dimensional space, such as along Cartesian x-, y-, and z-coordinates). As used herein, the term “orientation” refers to the rotational placement of an element or a portion of an element (c.g, three degrees of rotational freedom in three-dimensional space, such as about roll, pitch, and yaw axes, represented in angle-axis, rotation matrix, quaternion representation, and / or the like). As used herein, and for a device with a kinematic series, such as with a repositionable structure with a plurality of links coupled by one or more joints, the term “proximal” refers to a direction toward a base of the kinematic series, and “distal” refers to a direction away from the base along the kinematic series.

[0029] As used herein, the term “pose” refers to the multi-degree of freedom (DOF) spatial position and orientation of a coordinate system of interest attached to a rigid body. In general, a pose includes a pose variable for each of the DOFs in the pose. For example, a full 6-DOF pose for a rigid body in three-dimensional space would include 6 pose variablescorresponding to the 3 positional DOFs (e.g., x, y, and z) and the 3 orientational DOFs (e.g., roll, pitch, and yaw). A 3-DOF position only pose would include only pose variables for the 3 positional DOFs. Similarly, a 3-DOF orientation only pose would include only pose variables for the 3 rotational DOFs. Further, a velocity of the pose captures the change in pose over time (e.g., a first derivative of the pose). For a full 6-DOF pose of a rigid body in three-dimensional space, the velocity would include 3 translational velocities and 3 rotational velocities. Poses with other numbers of DOFs would have a corresponding number of velocities translational and / or rotational velocities.

[0030] Aspects of this disclosure are described in reference to computer-assisted systems, which can include devices that are teleoperated, externally manipulated, autonomous, semiautonomous, and / or the like. Further, aspects of this disclosure are described in terms of an implementation using a teleoperated surgical system, such as the da Vinci® Surgical System commercialized by Intuitive Surgical, Inc. of Sunnyvale, California. Knowledgeable persons will understand, however, that inventive aspects disclosed herein may be embodied and implemented in various ways, including teleoperated and nonteleoperated, and medical and non-medical embodiments and implementations.Implementations on da Vinci® Surgical Systems are merely exemplary and are not to be considered as limiting the scope of the inventive aspects disclosed herein. For example, techniques described with reference to surgical instruments and surgical methods may be used in other contexts. Thus, the instruments, systems, and methods described herein may be used for humans, animals, portions of human or animal anatomy, industrial systems, general robotic, or teleoperated systems. As further examples, the instruments, systems, and methods described herein may be used for non-medical purposes including industrial uses, general robotic uses, sensing or manipulating non-tissue work pieces, cosmetic improvements, imaging of human or animal anatomy, gathering data from human or animal anatomy, setting up or taking down systems, training medical or non-medical personnel, and / or the like.Additional example applications include use for procedures on tissue removed from human or animal anatomies (with or without return to a human or animal anatomy) and for procedures on human or animal cadavers. Further, these techniques can also be used for medical treatment or diagnosis procedures that include, or do not include, surgical aspects.

[0031] Referring now to the drawings, in which like reference numerals represent like parts throughout the several views, FIG. 1 shows an example repositionable assembly 100, in accordance with one or more embodiments.

[0032] FIG. 1 illustrates an example repositionable assembly 100 of a computer-assisted system. The repositionable assembly 100 includes a proximal repositionable structure 106 and one or more distal repositionable structures 102, 104, according to various embodiments. As shown in FIG. 1, a repositionable assembly 100 includes, without limitation, a proximal repositionable structure 106 that physically supports one or more distal repositionable structures (e.g., a first distal repositionable structure 102 and a second distal repositionable structure 104). The first distal repositionable structure 102 is configured to support a first instrument 122, and the second distal repositionable structure 104 is configured to support a second instrument 124. In various embodiments, each of the proximal repositionable structure 106, the first and second distal repositionable structures 102, 104, and the first and second instruments 122, 124 can include any number of joints 144 of any type, and any number of links 142 of any geometry. While FIG. 1 shows a proximal repositionable structure 106 supporting two distal repositionable structures 102, 104 configured to support instruments 122 and 124, respectively, the proximal repositionable structure 106 can support any number of distal repositionable structures, and the distal repositionable structures 102, 104 can each support any number of instruments. Furthermore, the first instrument 122 and / or the second instrument 124 may include any number of links 142 and any number of joints 144. While FIG. 1 illustrates instrument 122 as an instrument with jawed end effectors and instrument 124 as an imaging instrument, such illustration is merely illustrative. In particular, the distal repositionable structures 102 and 104 may support any instrument type.

[0033] The combination of the first instrument 122 and the first distal repositionable structure 102 may have the same number of joints as the combination of the second instrument 124 and the second distal repositionable structure 104. Alternatively, as shown in FIG. 1, the combination of the first instrument 122 and the first distal repositionable structure 102 may have fewer joints than the combination of the second instrument 124 and the second distal repositionable structure 104.

[0034] In some embodiments, the motion of each of the proximal repositionable structure 106, first distal repositionable structure 102, first instrument 122 with jawed end effectors (not labeled), second distal repositionable structure 104, and second instrument 124 with an imaging device (not labeled) is relative to a corresponding reference. The corresponding references can be the same reference, or be different references. A reference can be, for example and without limitation, a reference point, a reference line or other geometric feature, a reference frame, etc. The reference can be fixed to the environment(e.g., a point, geometric feature, or frame of reference fixed to the environment or the earth; such a reference frame can be called a “world frame”), to a subject of a procedure such as a workpiece or part of a patient (e.g., a point, geometric feature, or frame of reference fixed to a patient feature and / or the like; such a reference frame can be called a “subject frame”), to a base of the robotic system e.g., a base point, a base plane, frame of reference of the repositionable assembly or some other part of the robotic system), and / or the like. In some embodiments, the motions of the proximal repositionable structure 106, first distal repositionable structure 102, first instrument 122, second distal repositionable structure 104, and / or second instrument 124 are determined, controlled, or sensed relative to a same reference, or to different references. For example, the motion of the proximal repositionable structure 106 could be relative to a world frame while the motion of the first distal repositionable structure 102 and first instrument 122 could be relative to a reference point or reference frame fixed to the proximal repositionable structure 106.

[0035] In the example shown in FIG. 1, the bases of first distal repositionable structure 102 and second distal repositionable structure 104 are attached to a distal portion of the proximal repositionable structure 106, and motion of the proximal repositionable structure 106 moves the bases of first distal repositionable structure 102 and second distal repositionable structure 104. This “caused motion” of the bases of first distal repositionable structure 102 and second distal repositionable structure 104 can move distal portions of first distal repositionable structure 102 and / or second distal repositionable structure 104, and of any instruments or other elements attached to first distal repositionable structure 102 and second distal repositionable structure 104. In some embodiments, a caused motion (e.g., of the first distal repositionable structure 102 and first instrument 122 or of the second distal repositionable structure 104 and second instrument 124) is determined, responded to, or sensed relative to the same or a different reference as used for the motion of the proximal repositionable structure 106.

[0036] FIG. 2A illustrates an example computer-assisted system 200, according to various embodiments. As shown in FIG. 2A, the computer-assisted system 200 includes, without limitation, a repositionable assembly 210 and a user input system 250. In a teleoperation scenario, an operator (not shown) uses the user input system 250 to operate the repositionable assembly 210, such as in a leader-follower configuration (also often called teleoperation configuration or master-slave configuration in industry) of the computer-assisted system 200. In the leader-follower configuration, the user input system 250 is theleader, and the repositionable assembly 210 is the follower of the leader-follower configuration.

[0037] The repositionable assembly 210 can be used to introduce a set of instruments (not shown here; discussed below with reference to FIGs. 4A and 4B) to a work site through a port or entry guide 230 (a cannula is shown) inserted in an aperture. In a medical scenario, the work site can be on or within a body cavity of a patient, and the aperture can be a minimally invasive incision or a natural body orifice. When used, the port or entry guide 230 can be free-floating, held in place by a fixture separate from the repositionable assembly 210, or held by an entry guide supporting link 222 or other part of the repositionable assembly 210. The distal portion of the proximal repositionable structure, also referred to as a manipulator-supporting link 224 is coupled to additional joints and links 214, 220 of the repositionable assembly 210, and these additional joints and links 214, 220 are mounted on a base 212.

[0038] In accordance with one or more embodiments, FIG. 2A depicts a set of distal repositionable structures 226 coupled to the manipulator-supporting link 224. Each of the distal repositionable structures 226 includes a carriage (or other instrument-coupling link) configured to couple to an instrument, and each of the distal repositionable structures 226 include one or more joint(s) that can be driven to move the carriage. For example, a distal repositionable structure 226 can include a prismatic joint that, when driven, linearly moves the carriage and any instrument s) coupled to the carriage. In some embodiments, this linear motion is along an insertion axis, as further described below with reference to FIGs. 4A and 4B. In this configuration the elements of the repositionable assembly 210 that are proximal to the distal repositionable structures 226 may be understood as the proximal repositionable structure as previously described in reference to FIG. 1.

[0039] In some embodiments, the additional joints and additional links 214 and 220 are used to position the port 230 at the aperture or another location. FIG. 2 A illustrates a prismatic joint for vertical adjustment (as indicated by arrow “A”) and a rotary joint for horizontal adjustment (as indicated by arrow “B”). In some embodiments, drivable joints are used to robotically pivot the port 230 (and the instruments disposed within it at the time) in yaw, pitch, and roll angular rotations about a remote center of motion as indicated by arrows C, D, and E, respectively. In some examples, the drivable joints of the plurality of joints of the repositionable assembly 210 provide redundant degrees of freedom, and coordinated motion of the drivable joints can cause part of the proximal and / or distal repositionable structure(s) to pivot about a software-centered remote center of motion (SWC). The locationof the SWC can be moved and is enabled by coordinated motion of the drivable joints. In some examples, the repositionable assembly 210 has a hardware remote center (HWC) that is maintained through the movement of one or more joints of the repositionable assembly 210 (e.g., a specific set of joints) irrespective of the motion of other joints. In some examples, the repositionable assembly 210 includes drivable joints that provide redundant degrees of freedom that enable the repositionable assembly 210 to pivot part of the proximal and / or distal repositionable structure about a point other than the HWC and can switch between pivoting about a HWC or about a SWC. In review, these joints providing adjustments A, B, C, D, and E may be considered part of the proximal repositionable structure as previously described in reference to FIG. 1.

[0040] In some embodiments, actuation of the degrees of freedom provided by joint(s) of the instrument(s) is provided by actuators disposed in, or whose motive force (e.g., linear force or rotary torque) is transmitted to, the instrument(s). Examples of actuators include rotary motors, linear motors, solenoids, and / or the like. The actuators drive transmission elements in the manipulators and / or in the instruments to control the degrees of freedom of the instrum ent(s). For example, the actuators can drive rotary discs of the distal repositionable structure(s) that couple with rotary discs of the instrument(s), where driving the rotary discs of the instruments drives transmission elements in the instrument that couple to move the joint(s) of the instrument, or to move the end effector(s) of the instrument, as further discussed below with reference to FIGs. 4A and 4B. Accordingly, the degrees of freedom of the instrument(s) are controlled by actuators that drive the instrument(s) in accordance with control signals determined based on inputs from the associated input devices (e.g., input devices 252 of the user input system 250). The control signals are determined in order to cause instrument motion or other actuation as indicated by movement of the input control devices or any other control signal. Furthermore, in some embodiments, appropriately positioned sensors, e.g., encoders, potentiometers, and / or the like, are provided to enable measurement of indications of the joint positions, or other data that can be used to derive joint position, such as joint velocity. The actuators and sensors are disposed in, transmit to, and / or receive signals from the distal repositionable structure(s) 226.

[0041] While a particular configuration of the repositionable assembly 210 is shown in FIG. 2A, those skilled in the art will appreciate that embodiments of the disclosure can be used with any design of repositionable assembly. For example, a repositionable assembly can have any number and any types of degrees of freedom, may or may not be configured tocouple to a port, use a port other than a cannula, and / or other configuration different from what is shown in FIG 2 A.

[0042] In the embodiments shown in FIG 2A, the user input system 250 includes one or more input devices 252 operated by the operator (not shown). The one or more input devices 252 are contacted and manipulated by the hands of the operator, with one input device for each hand. Examples of such hand-input-devices include any type of device manually operable by human user, e.g., joysticks, trackballs, button clusters, and / or other types of haptic devices typically equipped with multiple degrees of freedom. Additionally, in some embodiments, position, force, and / or tactile feedback devices (not shown) are employed to transmit position, force, and / or tactile sensations from the instruments back to the operator's hands through the input devices 252.

[0043] The input devices 252 are supported by the user input system 250 and are shown as mechanically grounded, and in other implementations may be mechanically ungrounded. An ergonomic support 256 is provided in some implementations. For example, FIG. 2A shows an ergonomic support 256 including forearm rests on which the operator may rest his or her forearms while manipulating the input devices 252. In some examples, the operator performs tasks at a work site near the repositionable assembly 210 during a medical procedure by controlling the repositionable assembly 210 using the input devices 252. In some examples, a single input device 252 is used to control the repositionable assembly 210. In some examples, two input devices 252 are virtually linked such that movement in one input device is mimicked in the other, and / or that the relative position and orientation of the two input devices 252 can control different degrees of freedom of the repositionable assembly 210 and supported elements. For example, two virtually linked input devices 252 can be rotated about a center point between the two input devices 252 (e.g., like the movement of a steering wheel or handlebar), and this rotation can be mapped (e.g., to control) a degree of freedom of the repositionable assembly 210 and / or supported elements (e.g., roll or rotation of an instrument).

[0044] In some implementations, the input devices 252 consist of a kinematic series, such as with a repositionable structure with a plurality of links coupled by one or more joints. In general, sensors (e.g., encoders) are coupled to the components or joints of an input device to detect the position (and / or velocity, acceleration) of the components or joints throughout their respective degrees of freedom. Thus, the pose of an input device can be defined by the position and / or orientation of each of its components or joints along with a knowledge of the geometry and disposition of any interconnecting links between joints of the input device.The pose of the input device (or information regarding the position and orientation of each of its components and / or joints can be stored in a variety of mathematical or computational data structures such as a tensor. An input device consisting of a kinematic series may be configured to translate or rotate according to one or more degrees of freedom.

[0045] A display unit 254 is included in the user input system 250. The display unit 254 displays images for viewing by the operator. The display unit 254 provides the operator with a view of the workspace with which the repositionable assembly 210 interacts. The view can include, for example, stereoscopic images or three-dimensional images to provide a depth perception of the workspace and the instrum ent(s) of the repositionable assembly 210 in the workspace. The display unit 254 can be moved in various degrees of freedom to accommodate the operator’s viewing position and / or to provide control functions. Where a display unit (such as the display unit 254 is also used to provide control functions, such as to command the repositionable assembly, the display unit also includes an input device (e.g., another input device 252).

[0046] When using the user input system 250, the operator can sit in a chair or other support in front of the user input system 250, position his or her eyes to see images displayed by the display unit 254, grasp and manipulate the input devices 252, and rest his or her forearms on the ergonomic support 256 as desired. In some implementations, the operator can stand at the workstation or assume other poses, and the display unit 254 and input devices 252 may differ in construction, be adjusted in position (height, depth, efc.), etc.

[0047] FIG. 2B illustrates an example system 270, according to various embodiments. The system 270 corresponds to the computer-assisted system 200 and includes one or more computing systems 272. A computing system 272 includes a processing system and is used to process input provided by the user input system 250, e.g., from the input device(s) 252 manipulated by an operator. In some embodiments, a computing system 272 is further used to provide an output, e.g., a video image to the display unit 274. Examples of display unit 274 include LCDs, LEDs, organic LED displays, projectors, etc. In some embodiments, one or more computing systems 272 are used to control the repositionable assembly 210.

[0048] In one or more embodiments, the computing system(s) 272 executes one or more control methods. The control methods include instructions for controlling one or more components of the repositionable assembly 210. In one or more embodiments, joint movements of the repositionable assembly 210 are controlled by one or more control methods driving one or more joints using actuators of the repositionable assembly 210, the joint movements being calculated by a processor of a processing system of the computingsystem(s) 272. The control methods process control signals from the user input system 250 or elsewhere, and / or sensor signals (e.g., positional encoder data from joint position sensors, image data from image instruments such as ultrasonic probes or cameras or endoscopes, and / or the like), to calculate commands for the joint actuators.

[0049] In some embodiments, the control methods perform at least some of the calculations of the joint commands using vectors and / or matrices, some of which have elements corresponding to positions, velocities, and / or forces / torques of the joints. The range of alternative joint configurations available to the control methods can be conceptualized as a joint space. For example, in some embodiments, the joint space has as many dimensions as the repositionable assembly has degrees of freedom, and a particular configuration of the repositionable assembly represents a particular point in the joint space, with each coordinate corresponding to a joint state of an associated joint of the repositionable assembly.

[0050] As used herein, the term “state” of a joint or multiple joints refers to the control variables associated with the joint or the multiple joints, respectively. For example, the state of an angular joint refers to the angle defined by that joint within its range of motion, and / or to the angular velocity (or speed or direction) of the joint. Similarly, the state of an axial or prismatic joint refers to the joint's axial or linear position, and / or to its axial or linear velocity (or speed or direction). While one or more of the control methods described herein include position controllers, they often also have velocity control aspects. Alternative embodiments can rely primarily or entirely on velocity controllers, force controllers, acceleration controllers, and / or the like without departing from the disclosure. Various aspects of control systems that can be used in such devices are described in U.S. Pat. No. 6,699,177, which is incorporated herein by reference. In general, as long as the movements described are based on the associated calculations, the calculations of movements of the joints and movements of an end effector described herein are performed using a position control technique, a velocity control technique, an acceleration control technique, a force or torque control technique, a combination of some or all of the foregoing, and / or the like.

[0051] In some embodiments, the control modes include one or more other types of control modes. For example, during a robotic task being performed under the control of input devices 252 operated by a user, various joints of the repositionable assembly can be commanded to a same position and controlled to maintain static positions. However, in another control mode, one or more of the joints can be commanded to be “floating,” and facilitate motion of that joint due to externally applied force. For example, a joint held in place by a brake can be floated by partially or entirely releasing the brake. An example ofsuch a joint includes a passive joint held in place by an electromagnetic brake. As another example, a joint driven by actuator(s) can be held in place by commanding the actuator(s) to hold the joint position and be floating by updating the command to the actuator(s) to the then-current position, velocity, and / or acceleration of the joint. As a result, a floating joint is readily reconfigured by an externally applied force or torque, without a control algorithm and / or a braking force seeking to counteract the reconfiguration caused by sufficient externally applied force or torque. In some embodiments, a floating joint is further controlled to exhibit other characteristics or provide additional responses, such as to provide a certain type or level of damping response. A floating joint can still be braked, actuated, or otherwise managed for friction or gravity compensation. Such compensation can be provided by, for example and without limitation, passive springs, actively driven actuators, and / or the like. Further, in some embodiments, joints that are not moved by actuators can still be gravity compensated, friction compensated, dampened, and / or the like by actuators.

[0052] In various embodiments, multiple different control modes are combined during operation of the repositionable assembly. For example, some joints could be controlled to maintain position and resist or rebound from attempted external articulation of those joints, while other joints could be controlled to be floating and facilitate external articulation of those other joints. Parameters such as joint position, velocity, or acceleration of the joints are detected by joint sensors. The sensor signals are used to provide kinematic information of the repositionable assembly.

[0053] The architecture of the control methods used for controlling the repositionable assembly can be of any appropriate form. As a specific example, the control architecture can be hierarchical, and could include a high-level controller and multiple joint controllers. A commanded movement is received by the high-level controller in, for example, a Cartesiancoordinate space (referred to herein as Cartesian-space). The commanded movement could be, for example, based on a movement command (e.g., in the form of a position and / or velocity) received from the user input system 250, or any other system that provides a movement command. The commanded movement is converted into commanded joint positions or joint velocities (e.g., linear or angular joint positions, linear or angular joint velocities). In some embodiments, the conversion is performed using an inverse kinematics algorithm. Subsequently, the joint controllers convert the received commanded joint positions or velocities into commanded currents to drive the actuators producing joint movements. The joint movements together produce a repositionable assembly movement that reflects the commanded movement.

[0054] In some embodiments, a joint controller controls a joint position. In some embodiments, the joint controller controls other variables such as joint velocity and / or joint force (linear force or angular torque). A joint controller receives a feedback signal in the form of a sensed joint state from an associated joint sensor, which it can use for closed-loop control. The sensed joint state includes, for example and without limitation, a joint position, a joint velocity (or component of velocity such as speed or direction), a joint acceleration (or component of acceleration), and / or the like, representing the joint movement. The sensed joint state is derived from the signals obtained from the joint sensor. A joint sensor can be, for example, an encoder, a potentiometer, an accelerometer, a hall effect sensor, and / or the like. In some embodiments, a state observer or estimator (not shown) is used. Each joint controller can implement any appropriate control scheme, such as a proportional integral derivative (PID), proportional derivative (PD), full state feedback, sliding mode, and / or various other control schemes, without departing from the disclosure.

[0055] In one or more embodiments, the control methods further perform at least one of the steps described in FIG. 9 below.

[0056] A computing system 272 may include, without limitation, one or more computer processors, non-persistent storage (e.g., volatile memory, such as random access memory (RAM), cache memory), persistent storage (e.g., a hard disk, an optical drive such as a compact disk (CD) drive or digital versatile disk (DVD) drive, a flash memory, etc.), a communication interface (e.g., Bluetooth interface, infrared interface, network interface, optical interface, efc.), and / or numerous other elements and functionalities. In some embodiments, a computer processor of a computing system 272 is an integrated circuit for processing instructions. For example, the computer processor can be one or more cores or micro-cores of a processor.

[0057] In some embodiments, a communication interface of a computing system 272 includes an integrated circuit for connecting the computing system 272 to a network (not shown) and / or to another device, such as another computing system 272. Further, in some embodiments, the computing system 272 includes one or more output devices, such as a display unit 274, a printer, a speaker, external storage, or any other output device. Software instructions in the form of computer readable program code to perform embodiments of the disclosure are stored, in whole or in part, temporarily or permanently, on non-transitory computer readable medium. Specifically, the software instructions correspond to computer readable program code that, when executed by a processor(s), is configured to perform one or more embodiments of the disclosure. In some embodiments, a computing system 272 isconnected to or configured to be a part of a network. The network may include multiple nodes. Each node corresponds to a different computing system, group of computing systems, group of nodes, and / or the like.

[0058] In some embodiments, the repositionable assembly 210 couples to an instrument when used to perform a procedure. The instrument can include an imaging device, e.g., an endoscope or an ultrasonic probe, usable to capture images of the workspace and output the captured images to an auxiliary system 280. In some embodiments, the auxiliary system 280 processes the captured images using one or more image processing techniques prior to any subsequent display. For example, the auxiliary system 280 can overlay the captured images with a virtual control interface prior to displaying the combined images to the operator via the user input system 250. In some embodiments, one or more separate display units 274 are coupled with a computing system 272 and / or the auxiliary system 280 for local and / or remote display of images, such as images of the procedure site or other related images.

[0059] FIG. 3 illustrates another example repositionable assembly 300 of a computer-assisted system such as a robotic system, according to various embodiments. As shown in FIG. 3, a repositionable assembly 300 includes, without limitation, a proximal repositionable structure 306 that forms a common mechanical base for a first distal repositionable structure 312 and a second distal repositionable structure 314. The first distal repositionable structure 312 supports a first instrument 302, and the second distal repositionable structure 314 supports a second instrument 304. Each of the first distal repositionable structure 312, second distal repositionable structure 314, proximal repositionable structure 306, the first instrument 302, and second instrument 304 includes any number of joints (e.g., joints 344 and 346) of any type and / or any number of links (e.g., links 342) of any geometry.

[0060] As shown in FIG. 3, the first instrument 302 is equipped with a first end effector 322 and the second instrument 304 is equipped with a second end effector 324. As described below, various types of end effectors can be used. In some implementations, the end effector is considered part of an instrument. In such instances, an instrument can be defined or categorized based on its end effector.

[0061] As shown in FIG. 3, the proximal repositionable structure 306 provides a movable support for the first distal repositionable structure 312 and second distal repositionable structure 314. Further, in the example of FIG. 3, the first distal repositionable structure 312 and second distal repositionable structure 314 include prismatic joints 346 that enable translational movement of the first instrument 302 and the second instrument 304relative to the proximal repositionable structure 306. While two distal repositionable structures (the first distal repositionable structure 312 and the second distal repositionable structure 314) are shown in FIG. 3, any number of distal repositionable structures can be included in the repositionable assembly 300.

[0062] In some embodiments, the repositionable assembly 300 is part of a medical robotic system. For example, the repositionable assembly 300 can be configured as a tableside-installed medical robotic system. The proximal repositionable structure 306 could be attached to a base of a surgical or examination table. Further, the medical robotic system could include one or more additional sets of drivable joints with the same or a different design. For example, the repositionable assembly 300 can be installed on one side of the table, and a same or different repositionable assembly can be installed on the same side, or another side, of the table.

[0063] FIG. 4A illustrates an example instrument 400 (also referred to herein as an instrument 400), according to various embodiments. The instrument 400 in Figure 4A includes, without limitation, a shaft 410, and an end effector located at a first end of the instrument 400. A housing 430, arranged to releasably couple the instrument 400 to a distal repositionable structure (shown, for example, in FIG 2A), is located at a second end of the instrument 400. In some embodiments, the shaft 410 is rotatably coupled to the housing 430 to enable angular displacement of the shaft 410 relative to the housing 430, as indicated by arrows 448. In contrast, in some embodiments, the shaft 410 is not rotatably coupled to the housing 430 and the shaft 410 cannot be angularly displaced relative to the housing 430.

[0064] Various types of end effectors 440 can be used. For example, the end effector 440 can include one finger, two fingers (e.g., jaws 442 that may open and close), or three or more fingers. Examples of end effectors include, but are not limited to, scissors, forceps, staplers, cutting and cautery instruments, and / or the like. As another example, an end effector can further include an imaging device, e.g., an endoscope or an ultrasonic probe, to capture images of the workspace.

[0065] In some embodiments, an end effector 440 is actuated by transmission elements (e.g., cables, metal bands, screws, tubes, push rods, etc.) that connect parts of the instrument to drive elements (e.g., pulleys, capstans, spools, nuts, linear slides, or the like) (not shown) in the housing 430. Movement e.g., translation or rotation) of the drive elements thereby controls the position of the end effector, or other degrees of freedom such as jaw opening, such that the end effector may translate or rotate, the jaws may open and close, and / or the like. In some embodiments, upon coupling of an instrument 400 on a distalrepositionable structure, the drive elements engage with actuators of the distal repositionable structure, such as by engaging with transmission elements coupled to the actuators. As an example, a description of the control of an instrument like the instrument 400 can be found in U.S. Pat. No. 6,394,998, entitled “Surgical Tools for Use in Minimally Invasive Telesurgical Applications,” which is incorporated herein by reference.

[0066] In the example shown in FIG. 4 A, the joints of the instrument 400 include, without limitation, a wrist 420 proximal to the end effector 440 and two shaft offset joints 422, 424 proximal to the wrist 420. The wrist 420 may enable rotation of the end effector 440 in one or more directions. The shaft offset joints 422, 424 can enable, for example, a translational offset 426 of the end effector 440 relative to the insertion axis 412 using the additional link serially coupled between the instrument shaft 410 and the end effector 440, in addition to the rotating provided by the wrist 420. The shaft offset joints 422, 424 may, thus, increase the workspace reachable by the end effector 440 of the instrument 400. Like the end effector 440, the wrist 420 and the shaft offset joints 422, 424 may be actuated by control cables.

[0067] FIG. 4B illustrates an example instrument or instrument 450, according to various other embodiments. Instruments 400 and 450 can be used for robotic procedures such as robotic medical procedures (e.g., surgeries), in accordance with one or more embodiments.

[0068] The instrument 450 shown in FIG. 4B includes, without limitation, various elements of the instrument 400 shown in FIG. 4A and operates in a substantially similar manner to the instrument 400 shown in FIG. 4A. Specifically, the instrument 450 includes a shaft 460 and a wrist 470 proximal to an end effector 490. Further, the instrument 450 has an insertion axis 462 for insertion / retraction of the instrument 450. The instrument 450 also allows angular displacement of the shaft 460 relative to the housing 480 as indicated by the arrows 498. Unlike the instrument 400 in FIG. 4A, the instrument 450 is not equipped with shaft offset joints. Accordingly, the instrument 450 cannot achieve a translational offset of the end effector 440 relative to the insertion axis 462 as can the instrument 400. For example, the shaft 460 without shaft offset joints could be made more rigid, could be configured to allow the transmission of higher forces or torques, could be configured to transmit forces and torques with reduced friction, and / or the like, compared to a similar shaft that includes shaft offset joints (e.g., shaft 410). An example of an instrument that generally utilizes transmission of a higher forces compared to other instruments is a tissue stapler (“stapler”). Additionally, in various implementations, an instrument without shaft offsetjoints could be less costly, easier to service, maintain and / or clean than a comparable instrument with shaft offset joints. An instrument without shaft offset joints may be said to be “joggle-less” or have a reduced number of degrees of freedom relative to an instrument with shaft offset joints.

[0069] While FIGs. 4A and 4B show particular configurations of instruments, designed to engage with a particular type of distal repositionable structure, other configurations of instruments are within the scope of the disclosure. For example, embodiments of instruments 400 and 450 could have multi-degree-of-freedom wrists (e.g., pitch and yaw degrees of freedom), single-degree-of-freedom wrists (e.g., pitch or jaw), or no wrists. Also, various embodiments of instruments could have any suitable type of end effector including, for example, scissors, forceps, staplers, irrigation nozzles, hooks, scissors, blunt dissection instruments, needle drivers, imaging devices, and / or the like. Further, different housings can also be used to interface with different types of distal repositionable structure or different structures altogether e.g., direct interface with the proximal repositionable structure).

[0070] The subsequently discussed figures illustrate a repositionable assembly that is controlled by a computer-assisted system to move or reposition the distal portion of a proximal repositionable structure. For example, the distal portion of the proximal repositionable structure can be repositioned about a remote center of motion. As another example, the distal portion of the proximal repositionable structure can be moved along an insertion axis of the repositionable assembly to alter the relative position of the distal portion to, say, an aperture. A prismatic joint of a distal repositionable structure can be actuated to move or reposition the distal repositionable structure and / or an attached instrument, e.g., in response to movement of the distal portion along the insertion axis.

[0071] The distal portion of the proximal repositionable structure can support, or serve as a common mechanical base (e.g., manipulator-supporting link) for one or more distal repositionable structures each configured to support one or more instruments. Thus, movement or repositioning of the distal portion of the proximal repositionable structure can move, or affect, supported instruments. Movement or repositioning of the distal portion of the proximal repositionable assembly can be effectuated through a reconfiguration of the repositionable assembly. For example, the state of one or more joints of the repositionable assembly can be adjusted (i.e., a reconfiguration) to reposition the distal portion of the proximal repositionable structure. Similarly, use of one or mor degrees of freedom of a distalrepositionable structure can be considered under the configuration or reconfiguration of the repositionable assembly.

[0072] In some instances, the position and orientation of an instrument is maintained with respect to a world frame ( / .< ., using a reference fixed to the environment), e.g., through use of degrees of freedom of the instrument, in view of a reconfiguration of the repositionable assembly. An instrument with an end effector that has a position (and, in some instances, an orientation) that is maintained within or relative to a workspace ( / .< ., relative to a world frame) in view of other movements (e.g., repositioning of the proximal repositionable structure) may be said to be “anchored.” As an example, methods for coordinating movements of multiple instruments of a repositionable assembly are described in further detail in International Patent Publication No. WO2022 / 046787A1, entitled “Method and System for Coordinated Multiple-Tool Movement Using a Drivable Assembly,” which is incorporated herein by reference. Anchoring of an instrument can include a change to one or more degrees of freedom or joints of the instrument. That is, during a reconfiguration of the repositionable assembly (e.g., repositioning the distal portion of a proximal repositionable structure) an anchored instrument can execute compensatory adjustments to maintain its position (and, in some instances, orientation) in the workspace ( / .< ., relative to a world frame). In some instances, anchoring is achieved using an instrument equipped with shaft offset joints as described with respect to FIG. 4 A, where the shaft offset joints, in addition to other instrument joints such as a wrist and a rotatably coupled shaft, provide sufficient degrees of freedom to the instrument to maintain its position and orientation during reconfiguration of the repositionable assembly.

[0073] The subsequently discussed figures illustrate a repositionable assembly that is controlled by a computer-assisted system to move or reposition the distal portion of a proximal repositionable structure, for example, about a remote center of motion. Movement, or repositioning, of the distal portion of the proximal repositionable structure can move, or affect, supported instruments. For example, in some scenarios, a supported instrument (e.g., imaging device, stapler, etc. is repositioned and / or reoriented by reconfiguring the repositionable assembly, such as by repositioning the distal portion of the proximal repositionable assembly. Repositioning and / or reorienting of a supported instrument by repositioning the distal portion of the proximal repositionable assembly can be executed to maintain a remote center of the repositionable assembly. For example, the distal portion of the proximal repositionable assembly can be repositioned in a pivoting motion about theremote center where the pivoting motion, while maintaining the remote center, also serves to reposition and / or reorient one or more supported instruments.

[0074] The distal portion of the proximal repositionable structure can support, or serve as a common mechanical base (e.g., manipulator-supporting link) for one or more distal repositionable structures each configured to support one or more instruments. Thus, movement or repositioning of the distal portion of the proximal repositionable structure can move, or affect, supported instruments. Movement or repositioning of the distal portion of the proximal repositionable assembly can be effectuated through a reconfiguration of the repositionable assembly. For example, the state of one or more joints of the repositionable assembly can be adjusted (i.e., a reconfiguration) to reposition the distal portion of the proximal repositionable structure. Similarly, use of one or more degrees of freedom of a distal repositionable structure can be considered under the configuration or reconfiguration of the repositionable assembly.

[0075] Embodiments of the disclosure relate to a method to reposition a first instrument supported by a repositionable assembly, where the first instrument is or includes an imaging device, based on user input received with a user input system. In particular, embodiments of the disclosure relate to determining a reference frame within a workspace of the repositionable assembly and providing feedback to an operator at the user input system based on a change in the reference frame associated with the repositioning of the imaging device. As will be demonstrated, a stated benefit of providing feedback to an operator at the user input system based on a change in the determined reference frame, as opposed to an instrument frame at the imaging device, is that feedback resulting from kinematic differences of the input device(s) and the repositionable assembly is mitigated.

[0076] As shown in FIG. 5 A, a repositionable assembly includes a proximal repositionable structure, which is or includes a manipulator-supporting link 502 that supports multiple distal repositionable structures 550 and 552 coupled to multiple instruments 504 and 510. In other words, the manipulator-supporting link 502 forms a common mechanical base for the distal repositionable structures 550 and 552 that support the instruments 504 and 510. In some embodiments, the manipulator-supporting link 502 corresponds to the manipulatorsupporting link of a proximal repositionable structure as previously introduced with reference to FIG 2A. That is, in some embodiments, the manipulator-supporting link is considered the distal portion of a proximal repositionable structure. Hereafter, to be explicit, the distal repositionable structures 550 and 552 will be referred to as a first distal repositionable structure 550 and a second distal repositionable structure 552. The first distal repositionablestructure 550 can correspond to the first distal repositionable structure 102 and 312 of FIG. 1 and FIG. 3, respectively. Similarly, the second distal repositionable structure 552 can correspond to the second distal repositionable structure 104 and 314 of FIG. 1 and FIG. 3, respectively. Likewise, instruments 504 and 510 are hereafter referred to as an imaging device 504 and another instrument 510, where these instruments can correspond to the first instrument 122 and 302, and second instrument 124 and 304 of FIGs. 1 and 3, respectively. In the example of FIG. 5 A, the imaging device 504 is depicted with an imaging end effector (e.g., a camera) and the another instrument 510 is depicted with a jawed end effector.

[0077] In the example of FIG. 5 A, the imaging device 504 and another instrument 510 are inserted through a cannula 530 toward a workspace 598, and the cannula 530 is inserted through an aperture 592 in a barrier 594. In a medical scenario, the barrier 594 could be a body wall of a patient, and the aperture 592 could be a minimally invasive incision or a natural body orifice of the patient.

[0078] In one or more embodiments, the proximal repositionable structure, one or more distal repositionable structures, or one or more portions thereof, can define an insertion axis 590. That is, the insertion axis 590 can be described as a feature of, or defined by, the repositionable assembly. Movement of one or more joints of the proximal repositionable structure, for example, can result in movement of the manipulator-supporting link 502. When the manipulator-supporting link 502 is moved, the insertion axis 590 and portions of the first distal repositionable structure 550 and second distal repositionable structure 552 attached to the manipulator-supporting link 502 can also be moved. Such motion of the insertion axis 590, first distal repositionable structure 550, and second distal repositionable structure 552 caused by motion of the common mechanical base supporting the first and second distal repositionable structures 550 and 552 (e.g., the manipulator-supporting link 502) can cause motion of the first instrument 504 and the second instrument 510. For example, if the first distal repositionable structure 550 and second distal repositionable structure 552 are held fixed in configuration, then the movement of the manipulator-supporting link 502 also moves the first instrument 504 and the second instrument 510.

[0079] In one or more embodiments, the manipulator-supporting link 502 can be pivoted about a remote center of motion 540 (also called “remote center” 540). In the example shown in FIG. 5A the repositionable assembly has been positioned, and / or is controlled, such that the remote center of motion 540 is located approximately centrally in the aperture 592.1

[0080] In some embodiments, the proximal repositionable structure includes drivable joints that are used to pivot the manipulator-supporting link 502 about the remote center 540. Pivoting can consist of angular rotations in yaw, pitch, and roll directions with respect to the remote center 540. For example, FIG. 5A depicts a pitch direction and a roll direction indicated by the arrows D and E, respectively. In the depicted example, a yaw direction (not depicted) may result in a movement of the manipulator-supporting link 502 that is, at least initially, “into the page” or “out of the page.” Examples of drivable joints of the proximal repositionable structure that can be used to pivot the manipulator-supporting link 502 about the remote center 540 can be seen in FIG. 2A.

[0081] In some embodiments, the first distal repositionable structure 550 and second distal repositionable structure 552 include one or more joints or links used to insert or retract a supported instrument (e.g., imaging device 504 and another instrument 510) in the workspace 598 (e.g., prismatic joints). For example, FIG. 5A depicts insertion / retraction directions with arrows F and G for the first distal repositionable structure 550 and the second distal repositionable structure 552, respectively.

[0082] As previously discussed, instruments can have various degrees of freedom. In the example of FIG. 5A, the imaging device 504 and the another instrument 510 have shaft offset joints 525. Various instruments may comprise or lack joints for various degrees of freedom. For example, the imaging device 504 or the another instrument 510 could lack joints to support movement about a roll degree of freedom about a respective roll axis (e.g., insertion axis of the instrument). As another example, an instrument may not include an articulated wrist.

[0083] Under at least one control mode of the computer-assisted system (e.g., a camera control mode) one or more of the position and orientation of the imaging device 504 can be adjusted by reconfiguring the repositionable assembly. For example, in at least one control mode, reconfiguration of the repositionable assembly includes pivoting the manipulator-supporting link 502 of the proximal repositionable assembly about the remote center 540, inserting or retracting the imaging device 504 along its instrument insertion axis using the first distal repositionable structure 550, or a combination thereof.

[0084] In the depicted example of FIG. 5 A, the position and orientation of the imaging device 504 are realized, under a specified control mode (e.g., camera control mode), by pivoting the manipulator-supporting link 502 about the remote center 540 and inserting / retracting the imaging device 504 along an insertion axis coincident with its instrument shaft. Pivoting consists of angular rotations in yaw (not depicted), pitch (D), androll (E) directions. Insertion and retraction along the instrument shaft of the imaging device 504 is indicated with the arrow F. Thus, in the example of FIG. 5A, it may be said that the position and orientation of the imaging device 504 is controlled or determined using a yaw degree of freedom, a pitch degree of freedom, a roll degree of freedom, and an insertion (including insertion and retraction) degree of freedom of the repositionable assembly.Specifically, in the given example, the yaw degree of freedom, pitch degree of freedom, and roll degree of freedom are provided using one or more joints or links of the proximal repositionable structure and the insertion degree of freedom is provided by the first distal repositionable structure 550. Further, it may be said that the yaw degree of freedom, pitch degree of freedom, and roll degree of freedom are rotational degrees of freedom or can be classified as a type of degree of freedom relating to rotation ( / .< ., a rotation type degree of freedom) and the insertion degree of freedom is a translational degree of freedom (z.e., a translation type degree of freedom). Thus, in the depicted example of FIG. 5 A, the position and orientation of the imaging device 504 can be adjusted, under one or more control modes, by reconfiguring the repositionable assembly using the yaw, pitch, roll, and insertion degrees of freedom. Further, reconfiguration of the repositionable assembly to move, reposition, and / or reorient the manipulator-supporting link 502 and one or more supported instruments can be performed while maintaining the remote center 540.

[0085] Operation of the computer-assisted system can include the reception of a user input at a user input system including one or more input devices to reconfigure the repositionable assembly to reposition and / or reorient the imaging device 504 (e.g., pivoting the manipulator-supporting link 502 about the remote center 540 causing motion of the imaging device 504). Further, feedback can be provided at the one or more input devices. The feedback can include the restriction of motion of the one or more input devices based on a constraint in the reconfiguration of the repositionable assembly. For example, a degree of freedom of the repositionable assembly can reach a range of motion limit preventing further reconfiguration of the repositionable assembly with respect to, at least, that degree of freedom. As another example, the repositionable assembly may be constrained by an anchored instrument. For example, with reference to FIG. 5A, the another instrument 510 may be anchored such that the another instrument 510 executes compensatory adjustments to maintain its position (and, in some instances, orientation) in the workspace (z.e., relative to a world frame) in view of a reconfiguration of the repositionable assembly (e.g., repositioning of the distal portion of the proximal repositionable structure). These examples demonstrate instances where feedback can be provided at the one or more input devices where thisfeedback is indicative of a range of motion limit of one or more of the repositionable assembly and a supported instrument (e.g., an anchored instrument).

[0086] Conventionally, in situations where an operator would like to reposition a supported instrument such as the imaging device 504, motions of one or more input devices relative to an operator frame of reference (“operator frame”) can be used to determine corresponding motion commands for the repositionable assembly relative to an instrument frame of reference (“instrument frame”) of the imaging device 504. For example, a mapping, such as a rigid and scaled connection, between the operator frame and the instrument frame can be established. Under this conventional approach, control of the repositionable assembly using an input device is performed by executing joint commands in the repositionable assembly such that the instrument frame mimics the movement of the operator frame. An example of this conventional approach is illustrated using FIGs. 5A through 5D.

[0087] FIG. 5A depicts a conventional instrument frame 575. The instrument frame 575 is attached to a distal portion or end of the imaging device 504. The imaging device has a field of view axis 595 representing the center of the field of view (or an image) provided by the imaging device 504. The depicted instrument frame 575 is a Cartesian frame with three orthogonal axes, namely, a first instrument axis Xi, a second instrument axis yi, and a third instrument axis zi, where the subscript “i” indicates that the frame is an “instrument” frame. As seen in FIG. 5 A, the zi axis is colinear with the field of view axis 595. The instrument frame 575 is rigidly attached to the imaging device 504 such that movement of the imaging device 504 affects the instrument frame 575 and vice versa. In the context of FIGS. 5A-5D instrument frame 575 can be or correspond to an image frame of an image captured by the imaging device 504.

[0088] FIG. 5B depicts two input devices, namely, a first input device 562 and a second input device 564. The first and second input devices 562 and 564 may be equivalent to the input devices 252 of the user input system 250 depicted in FIG. 2B. In the example of FIG. 5B, the first and second input devices 562 and 564 are said to each consist of a kinematic series, such as with a repositionable structure with a plurality of links coupled by one or more joints, and can translate or rotate according to one or more degrees of freedom. For example, the first input device 562 can translate in one or more directions (e.g., three orthogonal directions) independent of the second input device 546. In general, sensors (e.g., encoders) are coupled to the components or joints of first and second input devices 562 and 564 to detect the position (and / or velocity, acceleration) of the components or joints throughout their respective degrees of freedom.

[0089] In the example of FIG. 5B, the first and second input devices 562 and 564 are virtually linked by a virtual link 566. The virtual link 566 acts to link or relate movement in one input device to the other, and / or to maintain the relative position and orientation of the first and second input devices 562 and 564. For example, a translation of the first input device 562 may require a translation of the second input device 564 to maintain a fixed distance between the first and second input devices 562 and 564 according to the virtual link 566. The virtual link 566 can be used to establish a shared operator frame 565. The depicted operator frame 565 is a Cartesian frame with three orthogonal axes, namely, a first operator axis x0, a second operator axis y0, and a third operator axis z0, where the subscript “o” indicates that the frame is an “operator” frame. In the example of FIG. 5B, the x0axis is colinear with the virtual link 566. Thus, coordinated motion of the first and second input devices 562 and 564 can be used to adjust the position and orientation of the operator frame 565 within an operator space 568. That is, the position and orientation of the operator frame 565 can change in the operator space 568 based on the relative positions of the first and second input devices 562 and 564 linked according to the virtual link 566. For example, the first and second input devices 562 and 564 can be rotated about the z0axis (e.g., like the movement of a steering wheel or handlebar) resulting in a rotation of the operator frame 565 about its z0axis.

[0090] In accordance with one or more embodiments, adjustments to the position and orientation of the operator frame 565 using the first and second input devices 562 and 564 are restricted to three translational degrees of freedom and one rotational degree of freedom. In the example of FIG. 5B, the operator frame 565 can translate within the operator space 568 along its three axes x0, y0, and z0, and rotate about its z0axis.

[0091] While FIG. 5B depicts a first input device 562 and a second input device 564 with a virtual link 566, where the relative positions of the linked first and second input devices 562 and 564 establish an operator frame 565 within an operator space 568, embodiments disclosed herein are applicable to other types of input device(s) and associated operator frames that may have different degrees of freedom. Examples of possible input devices include a trackpadjoystick, directional-pad, mouse, keyboard, trackball, etc.

[0092] FIG. 5C illustrates an adjustment to the position of the operator frame 565 in an operator space 568, where the operator frame is 565 is defined, similar to FIG. 5B, on a virtual link 566 of a virtually link first input device 562 and second input device 564. Thus, coordinated movement (including translation and rotation) of the first input device 562 and second input device 564, based on the virtual link 566 result in movement (includingtranslation and rotation) of the operator frame 565. In the example of FIG. 5C, the first input device 564 is translated from a first position to a second position where the second position of the first input device is labeled 562a. Similarly, the second input device 564 is translated from a first position to a second position where the second position of the second input device is labeled 564a. In this example, both the first input device 562 and the second input device 564 translate together in a “downward” motion within the depicted operator space (568), or along the depicted y0axis. The coordinated translation of the first and second input devices 562, 564 causes a translation of the operator frame 565 within the operator space 568 from a first position to a second position of the operator frame labeled 565a. The first position of first input device 562, second input device 564, and operator frame 565 are depicted in FIG.5C using solid lines and their respective second positions are depicted in FIG. 5C using dashed lines, in addition to the distinction in labels. As seen, in the example of FIG. 5C, the operator frame 565 is translated along its y0axis. As discussed above, under conventional approaches, control of the repositionable assembly using an input device is performed by executing joint commands in the repositionable assembly such that the instrument frame 575 (See FIG. 5 A) moves (translates and rotates) within a defined space (e.g., workspace 598) in a way that mimics the movement of the operator frame 565 in the operator space 568. For example, a mapping, such as a rigid and scaled connection, between the operator frame 565 and the instrument frame 575 can be established. Keeping with the example of FIG. 5C, under conventional methods a translation of the operator frame 565 from a first position to a second position (565 to 565a) along the y0axis of the operator frame 565 should result in movements (or commanded movements) of the repositionable assembly to translate the instrument frame 575 (FIG. 5A) along its yi axis. Similar statements can be made for translation along the respective x0, xi axes, z0, zi axes, and rotation about the z0, zi axes.

[0093] FIG. 5D illustrates an adjustment to the position and orientation of the operator frame 565 in an operator space 568 without regard for the configuration of the input device(s) used to realize this adjustment. A first position and orientation of the operator frame 565 is depicted in FIG. 5D using solid lines and a second position and orientation of the operator frame 565 is depicted in FIG. 5D using dashed lines. The second position and orientation of the operator frame is further labeled as 565a. As seen, the operator frame 565 has experienced a translation and a rotation in the operator space 568. In general, the change of the operator frame 565 is determined, or directly related to, a position, orientation, or input signal of one or more input devices. For example, a directional-pad used as an input device can be used by pressing and / or holding directions on the directional pad to translate theoperator frame 565 in the operator space 568. In some implementations, the operator space 568 is purely virtual and need not encompass a physical space, for example, enclosing one or more input devices.

[0094] As stated, a conventional approach for repositioning and / or reorienting the imaging device 504 is to relate changes to the operator frame of reference to the instrument frame. Changes to the operator frame are made using one or more input devices. For example, an objective of the conventional approach may be to determine corresponding motion commands for the repositionable assembly such that the position and orientation of the instrument frame in the workspace mimics or follows (e.g., with a scaling factor) the position and orientation of the operator frame in an operator workspace. For example, a translation of the operator frame based on the movement or input of one or more input devices results in a corresponding translation of the instrument frame and thus the imaging device 504 through a reconfiguration of the repositionable assembly.

[0095] However, in some situations, the instrument frame cannot move according to a movement of the operator frame. That is, the repositionable assembly cannot be reconfigured in such a manner as to move the instrument frame according to movements of the operator frame. Examples of such situations include instances where the degrees of freedom of the input device and the repositionable assembly are mismatched either in number of degrees of freedom or type of degrees of freedom. Thus, in these situations, feedback may be rendered to a user through the input device originating from kinematic differences between the input device and the repositionable assembly relative to their frames of reference rather than, for example, a constraint based on a range of motion limit of the instrument or repositionable assembly.

[0096] Following the examples of FIGs. 5A-5D, FIGs. 6A-6C provide an example an instrument frame being unable to mimic an arbitrary movement of an operator frame, under a conventional approach. FIG. 6A, identical to FIG. 5 A and reproduced to facilitate understanding of FIGs. 6B and 6C, depicts a repositionable assembly including a proximal repositionable structure. In particular, a manipulator-supporting link 502 is depicted. The manipulator-supporting link 502 supports a first distal repositionable structure 550 coupled with an attached imaging device 504 and a second distal repositionable structure 552 with an attached another instrument 510. The manipulator-supporting link 502 forms a common mechanical base for the distal repositionable structures 550 and 552 that support the instruments 504 and 510.

[0097] An insertion axis 590 of the proximal repositionable structure is defined. The manipulator-supporting link 502 can be pivoted about a remote center of motion 540. The manipulator-supporting link can further be rotated about the insertion axis 590, or other defined axis (e.g., an “anchoring axis” defined between the remote center 540 and a control point of the repositionable assembly, briefly described later in the instant disclosure). In the example shown in FIG. 6A the repositionable assembly has been positioned, and / or is controlled, such that the remote center of motion 540 is located approximately centrally in the aperture 592. In the depicted example of FIG. 6A, the proximal repositionable structure includes drivable joints that are used to pivot the manipulator-supporting link 502 about the remote center 540. Pivoting can consist of angular rotations in yaw, pitch, and roll directions with respect to the remote center 540. FIG. 6A depicts a pitch direction and a roll direction indicated by the arrows D and E, respectively. In the depicted example, a yaw direction (not depicted) may result in a movement of the manipulator-supporting link 502 that is, at least initially, “into the page” or “out of the page.” Additionally, the first distal repositionable structure 550 includes one or more joints or links used to insert or retract the supported imaging device 504 (e.g., prismatic joint). FIG. 6A depicts insertion / retraction directions with arrows F and G for the first distal repositionable structure 550 and the second distal repositionable structure 552, respectively.

[0098] In the example of FIG. 6A, the imaging device 504 is said to lack joints to support movement about a roll degree of freedom about its respective roll axis (e.g., insertion axis of the instrument). As another example, an instrument may not include an articulated wrist.

[0099] In the depicted example of FIG. 6 A, the position and orientation of the imaging device 504 are realized, under a specified control mode (e.g., camera control mode), by pivoting the manipulator-supporting link 502 about the remote center 540 and inserting / retracting the imaging device 504 along an insertion axis coincident with its instrument shaft. Pivoting consists of angular rotations in yaw (not depicted), pitch (D), and roll (E) directions. Insertion and retraction along the instrument shaft of the imaging device 504 is indicated with the arrow F. Thus, in the example of FIG. 6A, it may be said that the position and orientation of the imaging device 504 is controlled or determined using a yaw degree of freedom, a pitch degree of freedom, a roll degree of freedom, and an insertion (including insertion and retraction) degree of freedom of the repositionable assembly, where the yaw and pitch degrees of freedom provide a rotation movement (or pivoting movement) of the manipulator-supporting link 502 about the remote center 540. Similarly, the rolldegree of freedom is realized by a rotation of the manipulator-supporting link 502 about the insertion axis 590, or other axis such as an “anchoring axis,” as long as the remote center 540 is maintained during roll. Maintenance of, or respecting the constraint imposed by, the remote center 540 using the roll degree of freedom can be realized by rotating the manipulator-supporting link 502 about an axis that intersects the remote center 540. In the example of FIG. 6A, the insertion axis 590 intersects the remote center 540. In other implementations, an “anchoring axis” can be defined that intersects the remote center and use of the roll degree of freedom can occur about this axis. In the given example, the yaw degree of freedom, pitch degree of freedom, and roll degree of freedom are provided using one or more joints or links of the proximal repositionable structure and the insertion degree of freedom is provided by the first distal repositionable structure 550. Further, it may be said that the yaw degree of freedom, pitch degree of freedom, and roll degree of freedom are rotational degrees of freedom or can be classified as a type of degree of freedom relating to rotation (z.e., a rotation type degree of freedom) and the insertion degree of freedom is a translational degree of freedom (z.e., a translation type degree of freedom). Thus, in the depicted example of FIG. 6A, the position and orientation of the imaging device 504 can be adjusted, under one or more control modes, by reconfiguring the repositionable assembly using the yaw, pitch, roll, and insertion degrees of freedom. That is, in the depicted example of FIG. 6A, repositioning and / or reorientation of the imaging device 504 is performed, under one or more control modes (e.g., a camera control mode) using three rotational degrees of freedom and one translational degree of freedom.

[0100] FIG. 6A depicts a conventional instrument frame 575. The instrument frame 575 is attached to a distal portion or end of the imaging device 504. The imaging device 504 has a field of view axis 595 representing the center of the field of view (or an image) provided by the imaging device 504. The depicted instrument frame 575 is a Cartesian frame with three orthogonal axes, namely, a first instrument axis Xi, a second instrument axis yi, and a third instrument axis zi, where the subscript “i” indicates that the frame is an “instrument” frame. As seen in FIG. 6A, the zi axis is colinear with the field of view axis 595. The instrument frame 575 is rigidly attached to the imaging device 504 such that movement of the imaging device 504 affects the instrument frame 575 and vice versa. In the context of FIGS.6A-6C, instrument frame 575 can be or correspond to an image frame of an image captured by the imaging device 504.

[0101] FIGs. 6B and 6C depict two input devices, namely, a first input device 562 and a second input device 564 used to control the repositionable assembly partially depictedin FIG. 6A. In the given example, the first and second input devices 562 and 564 are virtually linked by a virtual link 566 that relates movement in one input device to the other, and / or to maintain the relative position and orientation of the first and second input devices 562 and 564. The virtual link 566 is used to establish a shared operator frame 565. Further, in this example, adjustments to the position and orientation of the operator frame 565 using the first and second input devices 562 and 564 are restricted to three translational degrees of freedom and one rotational degree of freedom. In the example of FIGs. 6B and 6C, the operator frame 565 can translate within the operator space (not depicted in FIGs. 6B and 6C) along its three axes x0, y0, and z0, and rotate about its z0axis.

[0102] FIG. 6B depicts a desired movement (e.g., a movement that a user of the user input system wants to realize) of the input devices 562, 564 to rotate the operator frame 565 about its z0axis. To be explicit, FIG. 6B depicts a coordinated (using the virtual link 566) rotation of the first input device 562 from a first position depicted with solid lines to a second position depicted with dashed lines and labeled 562a and the second input device 564 from a first position depicted with solid lines to a second position depicted with dashed lines and labeled 564a. The depicted rotation of the first and second input devices 562, 564, if realized, results in a rotation of the operator frame 565 about its z0axis from a first orientation to a second orientation, the second orientation distinguished using dashed lines and the label 565a.

[0103] Under a conventional method and a specified control mode (e.g., a camera control mode), given an input from a user as that depicted in FIG. 6B, the repositionable assembly should be reconfigured - moving through one or more of its pitch, yaw, roll, and insertion degrees of freedom while respecting the remote center 540 - to rotate the instrument frame about its corresponding zi axis. That is, in this example, a rotation of the operator frame 565 about its z0axis is intended to cause rotation of the instrument frame 575 about its zi axis. However, in this example, the instrument frame 575 - using the described pitch, yaw, roll, and insertion degrees of freedom the repositionable assembly - is incapable of rotating about its zi axis. This is because the instrument frame 575, using the described degrees of freedom of the repositionable assembly, can only rotate about an axis that intersects the remote center 540 and zi axis of the instrument frame 575, in the depicted example, does not pass through the remote center 540.

[0104] In other scenarios, a desired movement indicated using one or more input devices may not be realized at the instrument because the types of degrees of freedom for the movement of the operator frame 565 and the instrument frame 575 are mismatched. As seenin the given examples, movement (repositioning and reorientation) of the instrument frame 575 is realized using three rotational degrees of freedom and one translational degree of freedom (while respecting the remote center 540) and movement (repositioning and reorientation) of the operator frame 565 makes use of three translational degrees of freedom and one rotational degrees of freedom.

[0105] Inspection of the example of FIG. 6A reveals that rotation of the instrument frame 575 about its zi axis is approximated by rolling the imaging device 504 (and thus the instrument frame 575) about the insertion axis 590 (or other specified axis such as an anchoring axis that intersects the remote center 540). FIG. 6C depicts the movement of the first and second input devices 562 and 564 that result in a roll of the imaging device 504 about the insertion axis 590.

[0106] As discussed, in some implementations, feedback can be provided at the input devices 562, 564. The feedback can include the restriction of motion of the input devices 562, 564 based on a constraint in the reconfiguration of the repositionable assembly. For example, a degree of freedom of the repositionable assembly can reach a range of motion limit preventing further reconfiguration of the repositionable assembly with respect to, at least, that degree of freedom. In a described scenario, a user may attempt to rotate the view of the imaging device 504 by rotating the operator frame 565 about its z0axis as depicted in FIG. 6B. However, because a corresponding motion of the instrument frame 575 is restricted, feedback rendered to the input devices 562, 564 may force a user to implement the movement of the input devices 562, 564 depicted in FIG. 6C. Thus, in this scenario, feedback is rendered to the user through the input devices 562, 564 originating from kinematic differences between the input device(s) and the repositionable assembly with respect to movement of their frames of reference rather than, for example, a constraint based on a range of motion limit of the instrument or repositionable assembly.

[0107] As discussed, embodiments of the disclosure relate to determining a reference frame within a workspace of the repositionable assembly and providing feedback to an operator at the user input system based on a change in the reference frame associated with the repositioning of the imaging device. As will be demonstrated, a stated benefit of providing feedback to an operator at the user input system based a change in the determined reference frame, as opposed to an instrument frame at the imaging device, is that feedback resulting from kinematic differences of the input device(s) and the repositionable assembly are mitigated.

[0108] Embodiments of the disclosure relate to a method for reconfiguring the proximal repositionable structure and one or more distal repositionable structures (e.g., the distal repositionable structure supporting an imaging device) to position and orient (or reposition or reorient) an imaging device in response to an input from the user input system. Embodiments of the disclosure further relate to rendering feedback to the user at the user input system based on a reference frame, described below.

[0109] In accordance with one or more embodiments, a reference frame - for example, a Cartesian frame having orthogonal x, y, and z axes - is established based on the repositionable assembly, e.g., at a distal end of the proximate repositionable structure or attached distal repositionable structure(s). For example, in some implementations, the reference frame is established along an insertion axis of the proximal repositionable structure (described with respect to FIG. 7A, below). In other implementations, the reference frame is established along an anchoring axis that joins a remote center of the repositionable assembly and an control point (described with respect to FIG. 8A, below).

[0110] Once established, the position and orientation of the reference frame is related to the imaging device (or its instrument frame), or portion thereof, through a set of relationships. For example, in some implementations, a first reference axis (e.g., z axis) of the reference frame is aligned with an insertion axis of the proximal repositionable structure, where the proximal repositionable structure possesses a degree of freedom to “roll” about the insertion axis. Alternatively, the reference frame is aligned with an anchoring axis where the proximal repositionable structure possesses one or more degrees of freedom to roll about the anchoring axis. In these cases, the set of relationships can specify that the first reference axis (e.g., z axis) of the reference frame and the insertion axis or anchoring axis are to remain aligned and that roll of the proximal repositionable structure is mimicked by a rotation or roll of the reference frame about its aligned first reference axis (e.g., z axis). As another example, which may be combined with the previous example, the set of relationships can specify that the reference frame can translate (or slide) along the insertion axis (or anchoring axis) while maintaining the position and orientation of the proximal repositionable structure and moving a specified distal repositionable structure that supports the imaging device (and thus a conventional instrument frame) using an insertion / retraction degree of freedom of the distal repositionable structure. That is, the position and orientation of the reference frame is associated with the position and orientation of the imaging device (or instrument frame of the imaging device) though the set of relationships. In some embodiments, the set of relationships is such that the reference frame is not rigidly attached to the imaging device,however, their respective movements are coupled (e.g., flexibly coupled). Examples of the coupled movements are shown with respect to FIGs. 7A-7D and 8A-8C.

[0111] Input from the input device controls the position and orientation of the reference frame and thus the position and orientation of the imaging device (and thus the proximal repositionable structure and distal repositionable structure) through the set of relationships. In some instances, the position and orientation of the reference frame is constrained by the achievable movement of the proximal and distal repositionable structures. In one or more embodiments, feedback is rendered (e.g., haptic forces) to the user at the user input system indicative of the constrained movement of the reference frame.

[0112] A stated benefit of embodiments of the disclosure is that non-intuitive feedback rendered at the user input system is minimized. The benefit is emphasized in instances where the type and / or number of degrees of freedom of the user input system and those used by the repositionable assembly to reposition and / or reorient the imaging device (under one or more control modes) are different (e.g., mismatched types such as translational and rotational, reduced number of degrees of freedom due to other system constraints such as anchoring an instrument, etc.). Thus, feedback rendered to a user at the user input system is useful in that it notifies the user of issues with the workspace (such as ROM limits or forces on tissue) rather than artifacts originating from different kinematics of the user input system and specified usage of degrees of freedom of the repositionable assembly (e.g., to respect a remote center).

[0113] As an example, in response to a first user input to cause a roll of an instrument frame of an instrument (e.g., imaging device), the control system can be configured to cause a roll motion of at least a portion of the repositionable assembly (e.g., manipulator-supporting link) about a roll axis. This roll motion is determined using, at least in part, a reference frame of the repositionable assembly that is based on the roll axis. That is, a first user input that is conventionally indicative of a roll of the instrument frame (or instrument) is effectuated by a roll motion about the roll axis of the repositionable assembly where this roll motion need not be, or include, a roll about an axis of the instrument frame. In other words, through use of the reference frame, the repositionable assembly can be controlled to roll about its roll axis (e.g., using the roll degree of freedom of the proximal repositionable structure) in response to a user input to roll the instrument (or the instrument frame of the instrument) where such a user input may conventionally indicate a roll about an axis of the instrument frame. In some instances, the roll axis is the insertion axis of the repositionable assembly. In other instances, the roll axis is an anchoring axis of the repositionable assembly. In one or moreimplementations, the roll axis (e.g., the insertion axis or an anchoring axis) is determined by the control system based on a second instrument supported by the repositionable assembly (e.g., based on a type of instrument for the second instrument). That is, the repositionable assembly can support a first instrument such as an imaging device (or including an imaging device) and a second instrument.

[0114] FIG. 7 A depicts the determination of a reference frame in a workspace, in accordance with one or more implementations. FIGs. 7B-7D depict reconfigurations of the repositionable assembly to reposition and / or reorient a supported imaging device, where the reconfigurations are depicted relative to FIG. 7A.

[0115] FIG. 7A depicts a repositionable assembly including a proximal repositionable structure. In particular, a manipulator-supporting link 502 is depicted. The manipulatorsupporting link 502 supports a first distal repositionable structure 550 coupled with an attached imaging device 504 and a second distal repositionable structure 552 with an attached another instrument 510. The manipulator-supporting link 502 forms a common mechanical base for the distal repositionable structures 550 and 552 that support the instruments 504 and 510.

[0116] An insertion axis 590 of the proximal repositionable structure is defined. The manipulator-supporting link 502 can be pivoted about a remote center of motion 540. The manipulator-supporting link 502 can further be rotated about the insertion axis 590. In the depicted example of FIG. 7 A, the proximal repositionable structure includes drivable joints that are used to pivot the manipulator-supporting link 502 about the remote center 540.Pivoting can consist of angular rotations in yaw, pitch, and roll directions with respect to the remote center 540. FIG. 7A depicts a pitch direction and a roll direction indicated by the arrows D and E, respectively. In the depicted example, a yaw direction (not depicted) may result in a movement of the manipulator-supporting link 502 that is, at least initially, “into the page” or “out of the page.” Additionally, the first distal repositionable structure 550 includes one or more joints or links used to insert or retract the supported imaging device 504 (e.g., prismatic joint). FIG. 7A depicts insertion / retraction directions with arrows F and G for the first distal repositionable structure 550 and the second distal repositionable structure 552, respectively.

[0117] In the example of FIG. 7A, the imaging device 504 is said to lack joints to support movement about a roll degree of freedom about its respective roll axis (e.g., insertion axis of the instrument). As another example, an instrument may not include an articulated wrist.

[0118] In the depicted example of FIG. 7 A, the position and orientation of the imaging device 504 are realized, under a specified control mode (e.g., camera control mode), by pivoting the manipulator-supporting link 502 about the remote center 540 and inserting / retracting the imaging device 504 along an insertion axis coincident with its instrument shaft. Pivoting consists of angular rotations in yaw (not depicted), pitch (D), and roll (E) directions. Insertion and retraction along the instrument shaft of the imaging device 504 is indicated with the arrow F.

[0119] FIG. 7A depicts a conventional instrument frame 575 and a determined reference frame 585. The instrument frame 575 is attached to a distal portion or end of the imaging device 504. The depicted reference frame 585 is a Cartesian frame with three orthogonal axes, namely, a first reference axis xr, a second reference axis yr, and a third reference axis zr, where the subscript “r” indicates that the frame is a “reference” frame. In accordance with one or more embodiments, the reference frame 585 is determined to reside at an arbitrary location along the insertion axis 590 and with its zraxis colinear with the insertion axis 590. A set of relationships, depicted with connecting line 586 in FIG. 7A, specifies the relationship between the position and orientation of the reference frame and the imaging device 504, e.g., a point on a distal portion of the imaging device (or its instrument frame). In the depicted examples of FIGs. 7A-7D, the set of relationships is such that the connecting line 586 can be viewed as a rigid attachment of the reference frame 585 and a point on the distal portion of the of imaging device 504. In instances where the set of relationships specifies a rigid attachment, the reference frame 585 and instrument frame (or a point on the distal portion of the imaging device 504) may be said to be rigidly coupled.

[0120] Using FIG. 7A as a refence for a starting position and / or configuration of the depicted portions of the repositionable structure and supported instruments, FIG. 7B depicts a repositioning of the imaging device 504 from a received input using the reference frame 585. In the example of FIG. 7B, a user input is received from a user input system including one or more input devices. In the example of FIG. 7B, the user input is received, or represented, as a movement of an operator frame 565 along its y0axis. An example of such an input is depicted in FIG. 5C. In accordance with one or more embodiments, based on the received user input, the repositionable assembly is reconfigured (e.g., repositioning the manipulatorsupporting link 502) to reposition and reorient the reference frame 585 similar to the operator frame 565 (e.g., with a scaling factor) and the imaging device 504 (and its instrument frame 575, if considered) is repositioned using the set of relationships (e.g., connecting line 586).

[0121] It is noted that control logic relating the operator frame 565 (or representation thereof) and the reference frame 585, such as a scaling factor, can include a reversal aspect (e.g., negative multiplier) such that movement of the operator frame 565 along a given axis results in movement of the reference frame 585 in the opposite (or negative direction) along its corresponding axis. This behavior can be applied to all subsequent examples without limitation. This behavior (e.g., negative multiplier) can allow a user to manipulate an image frame (e.g., an instrument frame of an imaging device) as if, for example, reading a newspaper. For example, under this viewpoint, an input device may be “pulled closer” to a user indicating a desired enlargement of an image which is achieved by inserting, or “pushing away,” the imaging device into the workspace.

[0122] Keeping with the example of FIG. 7B, the reference frame 585 is moved along its yraxis to mimic movement of the operator frame 565 along its y0axis. In the given example, this movement is achieved by pivoting the manipulator-supporting link 502 about the remote center 540 using the pitch degree of freedom, D. As seen in FIG. 7B, the imaging device 504 is repositioned based on the movement of the reference frame 585. In particular, the connecting line 588 (representing the set of relationships) maintains a rigid attachment between the reference frame 585 and a distal point on the imaging device 504. Further, in the example of FIG. 7B, the position and orientation of the another instrument 510 in the workspace 598 is maintained using the degrees of freedom of the another instrument 510 to compensate for the repositioning of the manipulator-supporting link 502. That is, in the example, the another instrument 510 is anchored. In other examples, the another instrument 510 may not be anchored and instead move during reconfiguration of the repositionable assembly. In the example of FIG. 7B, the another instrument 510 is anchored using the shaft offset joints 525a and wrist 525b (or wrist assembly) of the another instrument 510. In this example, the movement of the operator frame 565 is performed (or represented) as a translation using one degree of freedom and, using the reference frame 585, reconfiguration of the repositionable assembly consists of using one rotational degree of freedom (namely, the pitch degree of freedom).

[0123] Still using FIG. 7A as a refence for a starting position and / or configuration of the depicted portions of the repositionable structure and supported instruments, FIG. 7C depicts a repositioning of the imaging device 504 from a received input using the reference frame 585. In the example of FIG. 7C, a user input is received from a user input system including one or more input devices. In the example of FIG. 7C, the user input is received, or represented, as a rotation of an operator frame 565 along its z0axis. An example of such aninput is depicted in FIG. 6B. In accordance with one or more embodiments, based on the received user input, the repositionable assembly is reconfigured (e.g., repositioning the manipulator-supporting link 502) to reposition and reorient the reference frame 585 similar to the operator frame 565 (e.g., with a scaling factor) and the imaging device 504 (and its instrument frame 575, if considered) is repositioned using the set of relationships (e.g., connecting line 586). In the example of FIG. 7C, the reference frame 585 is rotated its zraxis to mimic the rotation of the operator frame 565 about its z0axis. In the given example, this movement is achieved by rolling the manipulator-supporting link 502 about the insertion axis 590 using the roll degree of freedom, E. In FIG. 7C, rotation of the manipulator-supporting link 502 about the insertion axis 590 is depicted as a roll motion 535.

[0124] As seen in FIG. 7C, the imaging device 504 is repositioned based on the movement of the reference frame 585. In particular, the connecting line 586 (representing the set of relationships) maintains a rigid attachment between the reference frame 585 and a distal point on the imaging device 504. Further, in the example of FIG. 7C, the position and orientation of the another instrument 510 in the workspace 598 is maintained using the degrees of freedom of the another instrument 510 to compensate for the repositioning of the manipulator-supporting link 502. That is, in the example, the another instrument 510 is anchored. In other examples, the another instrument 510 may not be anchored and instead move during reconfiguration of the repositionable assembly. In the example of FIG. 7C, the another instrument 510 is anchored using one or more of the shaft offset joints 525a and wrist 525b (or wrist assembly) of the another instrument 510 and a roll degree of freedom of the another instrument about its insertion axis (depicted with second rotation 545). In this example, the movement of the operator frame 565 is performed (or represented) as a rotation using one degree of freedom and, using the reference frame 585, reconfiguration of the repositionable assembly consists of using one rotational degree of freedom (namely, the roll degree of freedom). In the depicted example of FIG. 7C, a user input that can be represented as a rotation of an operator frame 565 about a specified axis (e.g., z0axis), such as the user input depicted in FIG. 6B, can cause a reconfiguration of the repositionable assembly without rendering feedback to a user representative of kinematic differences between the user input system and the repositionable assembly.

[0125] Still using FIG. 7A as a refence for a starting position and / or configuration of the depicted portions of the repositionable structure and supported instruments, FIG. 7D depicts a repositioning of the imaging device 504 from a received input using the reference frame 585. In the example of FIG. 7D, a user input is received from a user input systemincluding one or more input devices. In the example of FIG. 7D, the user input is received to retract the imaging device 504. This input can be represented, for example, as a translation of an operator frame 565 along its z0axis. In accordance with one or more embodiments, based on the received user input, the repositionable assembly is reconfigured (e.g., insertion / retraction of the first distal repositionable structure 550 that supports the imaging device 504) to reposition and reorient the reference frame 585 similar to the operator frame 565 (e.g., with a scaling factor) and the imaging device 504 (and its instrument frame 575, if considered) is repositioned using the set of relationships (e.g., connecting line 586). In the example of FIG. 7D, the reference frame 585 is translated about its zraxis to mimic the translation of the operator frame 565 about its z0axis. In FIG. 7D, the reference frame 585 is depicted as being translated along the insertion axis 590 ( / .< ., along the zraxis) from an original location displayed using a frame with dotted lines to a new location, where the distance between the original location and new location is A. As seen in FIG. 7D, the imaging device 504 is repositioned based on the movement of the reference frame 585. In particular, the connecting line 586 (representing the set of relationships) maintains a rigid attachment between the reference frame 585 and a distal point on the imaging device 504. this movement is achieved by retracting the first distal repositionable structure 550 using the insertion / retraction degree of freedom, F. In particular, the first distal repositionable structure 550 is retracted by a distance A.

[0126] In this example, the movement of the operator frame 565 is performed (or represented) as a translation using one degree of freedom and, using the reference frame 585, reconfiguration of the repositionable assembly consists of using one translation degree of freedom (namely, the insertion / retraction degree of freedom of the distal repositionable structure that supports the imaging device).

[0127] The examples of FIGs. 7A-7D demonstrate that embodiments disclosed herein may be said to lend a mapping of degrees of freedom between a user input system and a repositionable assembly each repositionable with different types and / or number of degrees of freedom. For example, a user input system can include two user input devices consisting of repositionable structures and virtually linked as depicted in FIG. 5B to form an operator frame movable in three translational degrees of freedom and one rotational degree of freedom. In this example, the operator frame can be said to translate along a first reference axis xr, a second reference axis yr, and a third reference axis zr, and further rotate about the zraxis. Further, a repositionable assembly can include a proximal repositionable structure with one or more drivable joints such that a distal portion of the proximal repositionable structurecan be pivoted about a remote center and rolled about an axis that intersects the remote center. The repositionable assembly can further include one or more distal repositionable structures attached to the distal portion of the proximal repositionable structure. Each of the one or more distal repositionable structures is configured to support one or more instruments. Further, a distal repositionable structure can translate along an insertion axis of the distal repositionable structure and / or supported instrument. Thus, in this example, the repositionable assembly can be said to provide three rotational degrees of freedom and one translational degree of freedom, namely, pitch, yaw, roll, and insertion / retraction degrees of freedom. This this example, under one or more control modes (e.g., a camera control mode), the user input system is used to receive user inputs to reposition and / or reorient a supported imaging device. Further, in this example the degrees of freedom of the user input system and the repositionable assembly are mismatched in terms of type; the user input system having three translational degrees of freedom and one rotational degree of freedom and the repositionable assembly having three rotational degrees of freedom and one translational degree of freedom. In accordance with one or more embodiments, repositioning and / or reorienting of the imaging device is performed by relating the imaging device (e.g., instrument frame such as an image frame, etc.) to a reference frame using a set of relationships (e.g., rigid attachment), and repositioning and / or reorienting the reference frame to mimic movement of an operator frame (or representation thereof). Using the reference frame in the examples of FIGs. 7A-7D, a mapping of the degrees of freedom of the user input system to the degrees of freedom of the repositionable assembly is as follows. Translation of the operator frame along its x0axis using the user input system corresponds to use of the yaw degree of freedom of the repositionable assembly. Translation of the operator frame along its y0axis using the user input system corresponds to use of the pitch degree of freedom of the repositionable assembly. Translation of the operator frame along its z0axis using the user input system corresponds to use of the insertion / retraction degree of freedom of the repositionable assembly. And rotation of the operator frame about its z0axis using the user input system corresponds to use of the roll degree of freedom of the repositionable assembly. Thus, in this example, use of the reference frame as described to reposition and / or reorient a supported imaging device uses a one-to-one mapping of degrees of freedom between the user input system and the repositionable assembly even though there is a mismatch between types of degrees of freedom between these repositionable structures. Thus, feedback rendered to a user at the user input system is useful in that it notifies the user of issues with the workspace (such as ROM limits) rather than artifacts originating from different kinematics of the userinput system and specified usage of degrees of freedom of the repositionable assembly (e.g., to respect a remote center).

[0128] As discussed, in some implementations the reference frame is established along an anchoring axis that joins a remote center of the repositionable assembly and a control point. FIG. 8A depicts the determination of a reference frame in a workspace, in accordance with one or more implementations. FIGs. 8B and 8C depict reconfigurations of the repositionable assembly to reposition and / or reorient a supported imaging device, where the reconfigurations are depicted relative to FIG. 8A.

[0129] FIG. 8A depicts a repositionable assembly including a proximal repositionable structure. In particular, a manipulator-supporting link 502 is depicted. The manipulatorsupporting link 502 supports a first distal repositionable structure 550 coupled with an attached imaging device 504 and a second distal repositionable structure 552 with an attached another instrument 510. The manipulator-supporting link 502 forms a common mechanical base for the distal repositionable structures 550 and 552 that support the instruments 504 and 510.

[0130] The imaging device 504 in FIG. 8A is similar to the instrument 400 as described in reference to FIG. 4A, and includes shaft offset joints. In contrast, the another instrument 510 in FIG. 8 A does not include shaft offset joints. The another instrument 510 has a wrist 520 that can provide articulation of its end effector in one or more degrees of freedom e.g., pitch and yaw). The another instrument 510 can be, for example, a tissue stapler (“stapler”). Additionally, in the example of FIG. 8A, the imaging device 504 does not have the ability to roll about its instrument shaft. In order to enable the imaging device 504 to reorient about a roll degree of freedom despite the imaging device 504 itself being unable to perform such a roll motion, a roll movement of the manipulator-supporting link 502 is performed using the proximal repositionable structure.

[0131] In accordance with one or more embodiments, an anchoring axis 515 is defined between a remote center 540 of the repositionable assembly and a control point 526. The position of the control point 526 is based on the another instrument 510. In the depicted examples of FIGs. 8A-8C, the control point 526 is determined to be at a center of the wrist 520 of the another instrument 510. The roll motion 535 of the manipulator-supporting link 502 is performed about the anchoring axis 515. The roll motion 535 changes the position and the orientation of the imaging device 504 as illustrated in FIG. 8B.

[0132] FIG. 8A depicts a determined reference frame 585. In accordance with one or more embodiments, the reference frame 585 is determined to reside at an arbitrary locationalong the anchoring axis 515 with its zraxis colinear with the anchoring axis 515. A set of relationships, depicted with connecting line 586 in FIG. 8 A, specifies the relationship between the position and orientation of the reference frame 585 and the imaging device 504, e.g., a point on a distal portion of the imaging device (or its instrument frame). As will be demonstrated, in the depicted example of FIG. 8C, the set of relationships is such that the connecting line 586 does not require a rigid attachment of the reference frame 585 and a point on the distal portion of the of imaging device 504. In instances where the set of relationships does not specify a rigid attachment, the reference frame 585 may be said to be flexibly coupled to the instrument frame (or a point on the distal portion of the imaging device 504).

[0133] Using FIG. 8A as a refence for a starting position and / or configuration of the depicted portions of the repositionable structure and supported instruments, FIG. 8B depicts a repositioning of the imaging device 504 from a received input using the reference frame 585. In the example of FIG. 8B, a user input is received from a user input system including one or more input devices. In the example of FIG. 8B, the user input is received, or represented, as a rotation of an operator frame 565 along its z0axis. An example of such an input is depicted in FIG. 6B. In accordance with one or more embodiments, based on the received user input, the repositionable assembly is reconfigured (e.g., repositioning the manipulator-supporting link 502) to reposition and reorient the reference frame 585 similar to the operator frame 565 (e.g, with a scaling factor) and the imaging device 504 (and its instrument frame 575, if considered) is repositioned using the set of relationships (e.g, connecting line 586). In the example of FIG. 8B, the reference frame 585 is rotated about its zraxis to mimic the rotation of the operator frame 565 about its z0axis. In the given example, this movement is achieved by rolling the manipulator-supporting link 502 about the anchoring axis 515. As depicted in FIG. 8B, the another instrument 510 (e.g., stapler) remains stationary ( / .< ., position and orientation are maintained) with respect to a world frame throughout the roll motion 535 despite the another instrument 510 not having shaft offset joints for anchoring. The maintenance of the position and orientation of the another instrument 510 is realized by executing the roll motion 535 about the anchoring axis 515 with compensatory adjustments made using a rotation 545 of the instrument shaft and articulation of the wrist 520 of the another instrument 510. Further, in this example, the set of relationships can be viewed as a rigid attachment between the reference frame 585 and a distal portion or point of the imaging device 504.

[0134] Still using FIG. 8A as a refence for a starting position and / or configuration of the depicted portions of the repositionable structure and supported instruments, FIG. 8Cdepicts a repositioning of the imaging device 504 from a received input using the reference frame 585. In the example of FIG. 8C, a user input is received from a user input system including one or more input devices. In the example of FIG. 8C, the user input is received to insert the imaging device 504. This input can be represented, for example, as a translation of an operator frame 565 along its z0axis. In accordance with one or more embodiments, based on the received user input, the repositionable assembly is reconfigured (e.g., insertion / retraction of the first distal repositionable structure 550 that supports the imaging device 504) to reposition and reorient the reference frame 585 similar to the operator frame 565 (e.g., with a scaling factor) and the imaging device 504 (and its instrument frame 575, if considered) is repositioned using the set of relationships (e.g., connecting line 586). In the example of FIG. 8C, the reference frame 585 is translated about its zraxis to mimic the translation of the operator frame 565 about its z0axis. In FIG. 8C, the reference frame 585 is depicted as being translated along the anchoring axis 515 ( / .< ., along the zraxis) from an original location displayed using a frame with dotted lines to a new location, where the distance between the original location and new location is A. As seen in FIG. 8C, the imaging device 504 is repositioned based on the movement of the reference frame 585. In particular, the imaging device 504 is repositioned using the insertion / retraction degree of freedom of the first distal repositionable structure 550, F, where the imaging device 504 is displaced (in the insertion direction) along its insertion axis by a distance A.

[0135] Notably, in the example of FIG. 8C, the insertion axis of the imaging device 504 (or direction of degree of freedom F) is not parallel to the anchoring axis 515. As a result, motion of the reference frame 585 and a distal portion of the imaging device 504 does not respect a rigid attachment. This is depicted in FIG. 8C, where the original connecting line 586a before movement of the reference frame 585 and imaging device 504 is displayed along with a new connecting line 586b between the reference frame 585 and imaging device 504 after the movement. While subtle, FIG. 8C depicts that the original connecting line 586a and the new connecting line 586b have different lengths and orientations indicating that the reference frame 585 and distal portion of the imaging device 504 are not rigidly attached.

[0136] As discussed, once determined, the position and orientation of the reference frame is related to the imaging device (or its instrument frame), or portion thereof, through a set of relationships. In accordance with one or more embodiments, a first reference axis (e.g., z axis) of the reference frame is aligned with a specified axis of the proximal repositionable structure, where the proximal repositionable structure can “roll” about the specified axis (e.g., insertion axis or anchoring axis of the repositionable assembly orproximal repositionable structure). In some implementations, the set of relationships specifies that translation of the reference frame about the aligned first reference axis (e.g., z axis) is “passed through” to the insertion / retraction degree of freedom of the distal repositionable structure that supports the imaging device. An example of this implementation is depicted in FIG. 8C. In other implementations, the set of relationships specifies that translation of the reference frame about the aligned first reference axis (e.g., z axis) is “projected to” or “projected from” the insertion / retraction degree of freedom of the distal repositionable structure that supports the imaging device. In these implementations, a displacement distance Axof the reference frame need not equal a displacement distance A2of the imaging device along their respective insertion / retraction axes. In instances where the aligned first reference axis (whether aligned with an insertion axis or an anchoring axis) is parallel to the insertion axis of the imaging device (or the insertion / retraction degree of freedom of the supporting distal repositionable structure), the reference frame and a distal portion of the imaging device (or its instrument frame) can be considered rigidly attached regardless whether the set of relationships specifies a “pass through” or a “projection” behavior. Under another viewpoint, it may be said that the set of relationships specifies a rigid attachment between the reference frame and a distal portion of the imaging device (or its instrument frame) except in instances where 1) a user input results in translation of the reference frame along its aligned first reference axis and 2) the aligned first reference axis is not parallel to the insertion / retraction degree of freedom associated with the imaging device. When these two conditions are met, the set of relationships can specify an exception to the rigid attachment, where the exception “passes through” or “projects” a displacement distance between the reference frame and imaging device with respect to their respective insertion / retraction axes.

[0137] It is noted that while FIGs. 7A-5D and 8A-6C use a two-dimensional view and depict the use of two instruments, embodiments of the instant disclosure are not limited to use in two-dimensional spaces or with only two instruments. In general, the reference frame 585 is determined in a three-dimensional space. Further, embodiments disclosed herein are readily applicable to scenarios where the repositionable assembly supports one instrument, two instruments, or three or more instruments.

[0138] While FIGs. 1, 2A-2B, 3, 4A-4B, 5A-5D, 6A-6C, 7A-7D, and 8A-8C show various configurations of components, other configurations may be used without departing from the scope of the disclosure. For example, various components may be combined to create a single component. As another example, the functionality performed by a singlecomponent may be performed by two or more components. While instruments are described as being supported by a robotic arm with an instrument holder, an instrument may be supported by any type of repositionable structure, without departing from the disclosure. Further, while components are often described in context of medical scenarios such as surgical scenarios, embodiments of the disclosure are equally applicable to other domains that involve robotic manipulation, e.g., non-surgical scenarios or systems, non-medical scenarios or systems, and / or the like.

[0139] Commands to reconfigure the repositionable assembly, for example, to determine and perform a motion of a proximal repositionable structure and or distal repositionable structure to reposition and / or reorient a supported imaging device, can be executed on one or more processors of the control system of the computer-assisted system.

[0140] Determinations related to the reconfiguration of the repositionable assembly can concurrently, or jointly, consider constraints that may be imposed on the computer-assisted system. For example, range of motion limits of one or more instruments, e.g., anchored instruments, can constrain motion of a proximal repositionable structure. In one or more implementations, reconfiguration of the repositionable assembly is prevented if the associated motion is to violate an imposed constraint.

[0141] In one or more implementations, feedback is provided to a user of the computer-assisted system in response to reaching a constraint, such as a range of motion limit of an anchored instrument. Feedback can be provided in a variety of ways including, but not limited to: providing an audio and / or visual alert such as a tone or displayed warning message; and rendering haptic feedback on an associated input device. In one or more implementations, upon reaching an imposed constraint, a user can override an imposed limit or termination of the motion that violates (or will violate) the imposed constraint. In one example, during execution of a motion to reconfigure the repositionable assembly, as indicated by a user input received using an input device of the computer-assisted system, a range of motion limit of an anchored instrument is reached. In response, haptic feedback is provided at the input device indicating the prevention of further reconfiguration of the repositionable assembly to the user (at least with respect to a limited degree of freedom). Further, the user can select an option to continue with the reconfiguration in violation of the constraint (i.e., instrument is not anchored). Selection of the option can be realized by the user pressing a button (e.g., in response to a displayed message presenting the option), maintaining the user input to execute the violating motion (e.g., resisting haptic feedback at the input device for a predefined period), or other similar mechanism.

[0142] Turning to FIG. 9, a flowchart in accordance with one or more embodiments is shown. The flowchart of FIG. 9 depicts a method 900 for computer-assisted systems. The method 900 may be used reconfigure a repositionable assembly of computer-assisted system to reposition or reorient a supported instrument configured as an imaging device (or imaging instrument) and to render feedback to a user input system in coordination with the reconfiguration. One or more of the steps in FIG. 9 may be performed by various components of systems, previously described with reference to any of the preceding figures. While these figures illustrate particular configurations of computer assisted systems, the method is equally applicable to other configurations. The method may be executed on one or more processors, e.g., of the control system of the computer-assisted system.

[0143] While the various steps in the flowchart are presented and described sequentially, one of ordinary skill will appreciate that some or all of the steps may be executed in different orders, may be combined or omitted, and some or all of the steps may be executed in parallel. Additional steps may further be performed. Furthermore, the steps may be performed actively or passively. For example, some steps may be performed using polling or be interrupt driven in accordance with one or more embodiments of the invention.

[0144] The subsequently described steps may be performed for repositionable assemblies as previously described or any other repositionable assemblies. In some embodiments, the repositionable assembly supports multiple instruments (e.g., two, three, or more instruments), each having multiple links. In some examples, a repositionable assembly may comprise a proximal repositionable structure and one or more distal repositionable structures. The one or more distal repositionable structures may each support or couple to one or more instruments. Examples of repositionable assemblies and instruments disposed on the repositionable assemblies include but are not limited to those previously discussed in reference to FIGS. 1, 2A-2B, 3, and 4A-4B. For example, the repositionable assembly can include a proximal repositionable structure with a distal portion and a first distal repositionable structure attached to the distal portion. The first distal repositionable structure can be configured to support the supported instrument or imaging device. The subsequently described steps may be used to reposition an imaging device supported by the repositionable assembly, where the degrees of freedom of the repositionable assembly used to reposition or reorient the imaging device differ in one or more of number or type with respect to the user input system. Further, the subsequently described steps may be used to provide feedback (e.g., render haptic feedback) at one or more input devices of the user input system where, using the described method, the feedback is expected to represent constraints (e.g., range ofmotion limits) and not originate from kinematic differences in the applied usage (e.g., available motion or degrees of freedom) of the repositionable assembly and user input system.

[0145] Turning to FIG. 9, Step 902 of the method 900 indicates that the subsequent steps apply to a computer-assisted system that includes a control system, a repositionable assembly configured to support one or more instruments including an imaging device, and a user input system. The control system is coupled to the repositionable assembly and the user input system.

[0146] In one or more implementations, the repositionable assembly includes a proximal repositionable structure with a distal portion and a first distal repositionable structure attached to the distal portion. The imaging device can be supported by the first distal repositionable structure. Further, in these implementations, the repositionable assembly can further include a remote center, where the distal portion of the proximal repositionable structure is configured, or constrained, to pivot about the remote center (e.g., pitch and yaw degrees of freedom). Further, in these implementations, the repositionable assembly can define a first axis about which the distal portion of the proximal repositionable structure can be configured, or constrained, to rotate (e.g., a roll degree of freedom). Thus, the proximal repositionable structure, under one or more control modes, can pivot and rotate about the remote center and first axis, respectively, using pitch, yaw, and roll degrees of freedom to reposition and / or reorient a supported imaging device. Further, in these implementations, the first distal repositionable structure can translate along an insertion axis of the supported imaging device (e.g., an insertion / retraction degree of freedom). Thus, the first distal repositionable structure, under one or more control modes, can reposition the supported imaging device. Motion of the distal portion of the proximal repositionable structure (e.g., pitch, yaw, and roll) and the first distal repositionable structure (e.g., insertion / retraction) can be used to reposition and reorient the supported imaging device under one or more control modes (e.g., a camera control mode) of the computer-assisted system.

[0147] In Step 904, a reference frame is determined (e.g., within a workspace of the repositionable assembly) with the control system. The reference frame is offset from the imaging device and associated instrument frame. That is, an imaging device may be said to have an instrument frame. The instrument frame is typically located at a distal end, or most distal joint, of the imaging device and moves with a field of view of the imaging device. One or more embodiments of the disclosure do not require recitation of an instrument frame. Inthese embodiments, the reference frame is said to be offset from the imaging device or offset from a distal portion of the imaging device. In one or more implementations, the reference frame is located on a first axis of the repositionable assembly. The first axis can be an insertion axis of the repositionable assembly or its proximal repositionable structure, if applicable, or an anchoring axis defined between a remote center of the repositionable structure and a control point in the workspace. In this latter case, the control point is determined based on another instrument supported by the repositionable assembly. For example, the control point can be determined to reside on the wrist assembly of the another instrument. In some implementations, an anchoring axis is defined in response to the another instrument being of a specified instrument type (e.g., a stapler), having a specified attribute (e.g., no shaft offset joints), and / or having satisfied one or more instrument state conditions (e.g., “closed,” “triggered / fired,” etc. . In one or more implementations, the reference frame has a first reference axis (e.g., a z axis) that is colinear with the first axis of the repositionable assembly on which the reference frame is located. In one or more implementations, a roll of the distal portion of the proximal repositionable structure about the first axis corresponds to a rotation of the reference frame about its colinear axis (first reference axis) (e.g., z axis). In one or more implementations, translation of the reference frame about the first axis corresponds with a movement of the first distal repositionable structure along its insertion / retraction degree of freedom.

[0148] In Step 906, a user input to reposition or reorient the imaging device is received at the user input system. In one or more implementations, the user input system includes one or more input devices. The one or more input devices can each be repositionable structures movable, by a user, over one or mor degrees of freedom (e.g., translatable in a three-dimensional space). In one more implementations, the user input system includes two input devices that are virtually linked and rotatable (e.g., in a plane) about a center point. In one or more implementations, the one or more input devices can be used to represent and manipulate an operator frame. In these implementations, user input received at the user input system can be used to control the repositionable assembly by commanding the repositionable assembly to reconfigure, within constraints such as respecting a remote center, to move the reference frame according to a movement of an operator frame, or representation thereof. For example, the reference frame can be moved, through reconfiguration of the repositionable assembly, to mimic movement of the operator frame.

[0149] In Step 908, a change in the reference frame is determined, with the control system, based on the received user input. For example, in one or more implementations, the user input is used by the control system to command a reconfiguration of the repositionable assembly to reposition or reorient the imaging device. Reconfiguration of the repositionable assembly can cause, or be caused by, a change in the reference frame. For example, the repositionable assembly can be reconfigured such that the reference frame moves according to the received user input (e.g., to mimic an operator frame or representation thereof). Thus, a change in the reference frame is determined by the control system where the change is based on the user input.

[0150] In Step 910, feedback is provided to the user input system based on the change in the reference frame. For example, a limit or inability for the reference frame to change with respect to one or more of position and orientation in the workspace can be rendered to one or more input devices of the user input system as haptic feedback. In one or more implementations, the reference frame is changed such that no haptic force is rendered at the user input system. That is, in one or more embodiments, a lack of feedback such as a lack of haptic force is still considered “providing” feedback to the user input system. The feedback can include the restriction of motion of the input devices of the user input system based on a constraint in the reconfiguration of the repositionable assembly. As described, a stated benefit of providing feedback to an operator at the user input system based a change in the determined reference frame, as opposed to an instrument frame at the imaging device, is that feedback resulting from kinematic differences of the user input system and the repositionable assembly are mitigated (e.g., differences in one or more of the number or type of degrees of freedom used by the user input system (e.g, to manipulate an operator frame or representation thereof) and the repositionable assembly (e.g, limited to pitch, yaw, roll, and insertion / retraction degrees of freedom).

[0151] In one or more implementations, the behavior of method 900 in FIG. 9 is implemented by the computer-assisted system under one or more specified control modes. The one or more control modes can include so-called camera control modes or control modes for control of a supported imaging device (e.g., to reposition and / or reorient) the imaging device. In one or more implementations, the computer-assisted system can detect an instrument type of each instrument attached to, or supported by, the repositionable assembly. In one or more implementations, the instrument type of the instrument is made known through specification of the end effector associated with the instrument (e.g., imaging device or camera). Further, in one or more embodiments, the one or more control modes can specifydegrees of freedom available for use to reposition and / or reorient a supported imaging device. For example, the one or more control modes can constrain reconfiguration of the repositionable assembly to respect a remote center. In one or more implementations, the one or more control modes indicate: 1) that the repositionable assembly is reconfigured, or to be reconfigured, to reposition and / or reorient a supported imaging device; and 2) the reconfiguration of the repositionable assembly consists of, or is limited to, pivoting and rotating of a distal portion of a proximal repositionable structure about a remote center and first axis, respectively (i.e., pitch, yaw, and roll degrees of freedom) and translating a first distal repositionable structure about its insertion axis ( / .< ., insertion / retraction degree of freedom).

[0152] Although only a few example embodiments have been described in detail above, those skilled in the art will readily appreciate that many modifications are possible in the example embodiments without materially departing from this invention. Accordingly, all such modifications are intended to be included within the scope of this disclosure as defined in the following claims.

Claims

CLAIMSWhat is claimed is:

1. A computer-assisted system comprising:a user input system configured to receive an input from a user;a repositionable assembly configured to support one or more instruments, the one or more instruments including an imaging device; anda control system comprising one or more processors and communicatively coupled to the user input system and the repositionable assembly, the control system configured to:determine a reference frame, the reference frame being offset from an instrument frame of the imaging device,receive, at the user input system, a user input to reposition or reorient the imaging device,determine, based on the user input, a change in the reference frame, and provide, based on the change in the reference frame, feedback to the user input system.

2. The computer-assisted system of claim 1, wherein the control system is further configured to:determine, based on the change in the reference frame, a motion to reposition or reorient the imaging device; andexecute the motion to reposition or reorient the imaging device.

3. The computer-assisted system of any of claims 1-2, wherein the repositionable assembly is associated with an insertion axis and the reference frame is based on the insertion axis.

4. The computer-assisted system of claim 1, wherein:the repositionable assembly comprises a proximal repositionable structure comprising a distal portion, andthe proximal repositionable structure is associated with a remote center.

5. The computer-assisted system of claim 4, wherein:the repositionable assembly further comprises a first distal repositionable structure attached to the distal portion of the proximal repositionable structure, and the imaging device is supported by the first distal repositionable structure.

6. The computer-assisted system of claim 5, wherein the control system is further configured to:determine, based on the change in the reference frame, a motion to reposition or reorient the imaging device; andexecute the motion to reposition or reorient the imaging device,wherein the motion corresponds to one or more of a reconfiguration of the proximal repositionable structure or an insertion or retraction of the first distal repositionable structure.

7. The computer-assisted system of claim 5, wherein:the repositionable assembly further comprises a second distal repositionable structure attached to the distal portion of the proximal repositionable structure, and the second distal repositionable structure supports another instrument.

8. The computer-assisted system of claim 7, wherein:an anchoring axis is based on the remote center and a control point based on the another instrument, andthe reference frame is located on the anchoring axis.

9. The computer-assisted system of claim 7, wherein the another instrument has an instrument type of a stapler.

10. The computer-assisted system of claim 7, wherein the another instrument does not have shaft offset joints.

11. The computer-assisted system of claim 7, wherein the control system is further configured to:determine, based on the change in the reference frame, a motion to reposition or reorient the imaging device using one or more of the proximal repositionable structure or the first distal repositionable structure;execute the motion to reposition or reorient the imaging device;determine a compensatory motion of the another instrument; andexecute the compensatory motion with the motion to reposition or reorient the imaging device.

12. The computer-assisted system of any of claims 1-2, wherein:the reference frame is a Cartesian frame comprising a first reference frame axis, the reference frame is located on a first axis of the repositionable assembly, and the first reference axis of the reference frame is colinear with the first axis.

13. The computer-assisted system of claim 12, wherein the first axis is an insertion axis or an anchoring axis of the repositionable assembly.

14. The computer-assisted system of claim 12, wherein:the repositionable assembly comprises a proximal repositionable structure comprising a distal portion and configured to roll the distal portion about the first axis and pivot the distal portion about a remote center,wherein the reference frame rotates about its first reference axis as the distal portion of the proximal repositionable structure rolls about the first axis.

15. The computer-assisted system of claim 14, wherein:the repositionable assembly further comprises a first distal repositionable structure attached to the distal portion of the proximal repositionable structure and the first distal repositionable structure supports the imaging device, the reference frame is configured to translate along the first axis,translation of the reference frame along the first axis corresponds to an insertion or retraction movement of the first distal repositionable structure.

16. The computer-assisted system of claim 15, wherein a distance of translation of the reference frame along the first axis is equal to a distance of translation of the first distal repositionable structure.

17. The computer-assisted system of any of claims 1-2, wherein:the user input system defines an operator frame and the user input determines a movement of the operator frame,the control system is further configured to determine a difference between the operator frame and the reference frame, andthe feedback provided to the user input system is based on the difference.

18. The computer-assisted system of any of claims 1-2, wherein:the user input system comprises a first input device configured to receive the user input,the feedback to the user input system comprises haptic feedback rendered to the first input device.

19. The computer-assisted system of claim 18, wherein:the user input system further comprises a second input device configured to receive, with the first input device, the user input,the first input device and the second input device are virtually linked and can be rotated about a center point between the first and second input devices, an operator frame is defined at the center point,the feedback to the user input system is based on a difference of the operator frame and the reference frame.

20. The computer-assisted system of any of claims 1-2, wherein the reference frame is rigidly coupled to the instrument frame.

21. The computer-assisted system of any of claims 1-2, wherein the reference frame is flexibly coupled to the instrument frame.

22. The computer-assisted system of any of claims 1-2, wherein the control system is configured to determine, based on a second instrument supported by the repositionable assembly, whether to rigidly couple the reference frame to the instrument frame or to flexibly couple the reference frame to the instrument frame.

23. The computer-assisted system of any of claims 1-2, wherein the reference frame is based on an insertion axis of the repositionable assembly.

24. The computer-assisted system of any of claims 1-2, wherein the reference frame is based on an anchoring axis that is offset from an insertion axis of the repositionable assembly.

25. The computer-assisted system of claim 24, wherein the anchoring axis is based on a remote center of the repositionable assembly and a control point associated with a second instrument supported by the repositionable assembly.

26. The computer-assisted system of claim 1, wherein:in response to a first user input to cause a roll of the instrument frame, the control system is configured to cause a roll motion of at least a portion of the repositionable assembly about a roll axis, andthe reference frame is based on the roll axis.

27. The computer-assisted system of claim 26, wherein the control system is configured to determine the roll axis based on a second instrument supported by the repositionable assembly.

28. The computer-assisted system of claim 26, wherein the roll axis is an insertion axis of the repositionable assembly.

29. The computer-assisted system of claim 26, wherein the roll axis is an anchoring axis that is offset from an insertion axis of the repositionable assembly.

30. A method for controlling a computer-assisted system, the method performed by a control system of the computer-assisted system and comprising:receiving a user input from a user input system comprised by the computer-assisted system, wherein:the computer-assisted system further comprises a repositionable assembly configured to support one or more instrument, the one or more instruments including an imaging device, andthe user input is indicative of a command to reposition or reorient the imaging device;determining a reference frame, the reference frame being offset from an instrument frame of the imaging device;determining, based on the user input, a change in the reference frame; and providing, based on the change in the reference frame, feedback to the user input system.

31. The method of claim 30, further comprising:determining, based on the change in the reference frame, a motion to reposition or reorient the imaging device; andexecuting the motion to reposition or reorient the imaging device.

32. The method of any of claims 30-31, wherein the repositionable assembly is associated with an insertion axis and the reference frame is based on the insertion axis.

33. The method of claim 30, wherein:the repositionable assembly comprises a proximal repositionable structure comprising a distal portion, andthe proximal repositionable structure is associated with a remote center.

34. The method of claim 33, wherein:the repositionable assembly further comprises a first distal repositionable structure attached to the distal portion of the proximal repositionable structure, and the imaging device is supported by the first distal repositionable structure.

35. The method of claim 34, further comprising:determining, based on the change in the reference frame, a motion to reposition or reorient the imaging device; andexecuting the motion to reposition or reorient the imaging device,wherein the motion corresponds to one or more of a reconfiguration of the proximal repositionable structure or an insertion or retraction of the first distal repositionable structure.

36. The method of claim 34, wherein:the repositionable assembly further comprises a second distal repositionable structure attached to the distal portion of the proximal repositionable structure, and the second distal repositionable structure supports another instrument.

37. The method of claim 36, wherein:an anchoring axis is based on the remote center and a control point based on the another instrument, andthe reference frame is located on the anchoring axis.

38. The method of claim 36, wherein the another instrument has an instrument type of a stapler.

39. The method of claim 36, wherein the another instrument does not have shaft offset joints.

40. The method of claim 36, further comprising:determining, based on the change in the reference frame, a motion to reposition or reorient the imaging device using one or more of the proximal repositionable structure or the first distal repositionable structure;executing the motion to reposition or reorient the imaging device;determining a compensatory motion of the another instrument; andexecuting the compensatory motion with the motion to reposition or reorient the imaging device.

41. The method of any of claims 30-31, wherein:the reference frame is a Cartesian frame comprising a first reference frame axis, the reference frame is located on a first axis of the repositionable assembly, and the first reference axis of the reference frame is colinear with the first axis.

42. The method of claim 41, wherein the first axis is an insertion axis or an anchoring axis of the repositionable assembly.

43. The method of claim 41, wherein:the repositionable assembly comprises a proximal repositionable structure comprising a distal portion and configured to roll the distal portion about the first axis and pivot the distal portion about a remote center,wherein the reference frame rotates about its first reference axis as the distal portion of the proximal repositionable structure rolls about the first axis.

44. The method of claim 43, wherein:the repositionable assembly further comprises a first distal repositionable structure attached to the distal portion of the proximal repositionable structure and the first distal repositionable structure supports the imaging device, the reference frame is configured to translate along the first axis,translation of the reference frame along the first axis corresponds to an insertion or retraction movement of the first distal repositionable structure.

45. The method of claim 44, wherein a distance of translation of the reference frame along the first axis is equal to a distance of translation of the first distal repositionable structure.

46. The method of any of claims 30-31, wherein:the user input system defines an operator frame and the user input determines a movement of the operator frame,the method further comprises determining a difference between the operator frame and the reference frame, andthe feedback provided to the user input system is based on the difference.

47. The method of any of claims 30-31, wherein:the user input system comprises a first input device configured to receive the user input, providing feedback to the user input system comprises rendering haptic feedback to the first input device.

48. The method of claim 47, wherein:the user input system further comprises a second input device configured to receive, with the first input device, the user input,the first input device and the second input device are virtually linked and can be rotated about a center point between the first and second input devices, an operator frame is defined at the center point,the feedback to the user input system is based on a difference of the operator frame and the reference frame.

49. The method of any of claims 30-31, wherein the reference frame is rigidly coupled to the instrument frame.

50. The method of any of claims 30-31, wherein the reference frame is flexibly coupled to the instrument frame.

51. The method of any of claims 30-31, further comprising determining, based on a second instrument supported by the repositionable assembly, whether to rigidly couple the reference frame to the instrument frame or to flexibly couple the reference frame to the instrument frame.

52. The method of any of claims 30-31, wherein the reference frame is based on an insertion axis of the repositionable assembly.

53. The method of any of claims 30-31, wherein the reference frame is based on an anchoring axis that is offset from an insertion axis of the repositionable assembly.

54. The method of claim 53, wherein the anchoring axis is based on a remote center of the repositionable assembly and a control point associated with a second instrument supported by the repositionable assembly.

55. The method of claim 30, further comprising:executing a roll motion of at least a portion of the repositionable assembly about a roll axis in response to a first user input to cause a roll of the instrument frame, wherein the reference frame is based on the roll axis.

56. The method of claim 55, further comprising determining the roll axis based on a second instrument supported by the repositionable assembly.

57. The method of claim 55, wherein the roll axis is an insertion axis of the repositionable assembly.

58. The method of claim 55, wherein the roll axis is an anchoring axis that is offset from an insertion axis of the repositionable assembly.

59. A non-transitory machine-readable medium comprising a plurality of machine-readable instructions executed by one or more processors associated with a computer-assisted system, the plurality of machine-readable instructions causing the one or more processors to perform the method of any of claims 30 to 55.

Images