Surgical performance automated assessment systems and methods

Abstract

A surgical performance automated assessment system is provided that includes a surgical trainer with a camera configured to capture video data of a surgical exercise and a surgical training evaluator operationally connected to the surgical trainer and configured to determine a state of a surgical activity for the surgical exercise or recognizing a predefined object of interest based analyzing the captured video data.

Description

Docket No.: 10133-USP-PCTSurgical Performance Automated Assessment Systems and MethodsCross-reference to Related Applications

[0001] This application claims priority to and benefit of U.S. Provisional Patent Application Serial No. 63 / 763,057, filed on February 25, 2025, U.S. Provisional Patent Application Serial No. 63 / 799,280, filed on May 2, 2025, U.S. Provisional Patent Application Serial No.63 / 887,383, filed on September 24, 2025, and U.S. Provisional Patent Application Serial No. 63 / 984,952, filed on February 17, 2026, all the disclosures of which are incorporated herein by reference in their entirety.Background

[0002] The present application generally relates to surgical training, and in particular, to surgical performance automated assessment training systems and methods.

[0003] Despite the continual rise in surgical care, many surgical procedures are rarely or lightly encountered during medical training. Also, there is an apparent growing concern amongst the surgical community regarding sufficient availability and exposure to surgical training.Challenges thus remain for trainees who desire accurate, repeatable, and accessible training prior to entering the operation room. Likewise, there are challenges for assessors of such trainees to access such trainee surgical performances as well as accurately and repeatably apply such assessments and further instruction and training. Furthermore, there is a need for precise and objective assessment methods in surgical training. Traditional evaluation techniques, often reliant on human observation and subjective judgment, can be prone to inconsistencies. Prior systems often suffer from objectivity, lacking consistent or fair evaluations, efficiency, lacking real-time analysis and feedback to expedite the learning process, allowing trainees to make immediate adjustments, or scalability, lacking versatility or applicability to a wide range of surgical tasks. Other systems also lack comprehensive performance data providing valuable insights for both trainees and instructors, facilitating targeted skill development. There is thus a need to address the challenges of assessing surgical performances.Docket No.: 10133-USP-PCTSummary

[0004] In accordance with various embodiments, a surgical performance automated assessment system is provided. In various embodiments, a surgical performance automated assessment system comprises a surgical trainer comprising a camera positioned to capture video data of a surgical exercise positioned within the surgical trainer and a surgical training evaluator.

[0005] In various embodiments, a surgical training evaluator comprises one or more processors and, in various embodiments, using segmentation, is configured to analyze captured video data including being configured to convert pixels from captured video data to a YUV color space in which each converted pixel is assigned a brightness component (Y), a first color projection component (U), and a second color projection component (V); identify one or more areas of interest (AOIs) in the converted pixels within a UV color space from the YUV color space, the UV color space comprising the first and second color projection components of the converted pixels and ignoring the brightness component, each AOI of the identified one or more AOIs corresponding to a predefined object of interest; generate a mask designating pixels in a frame of the captured video data that correspond to an AOI of the identified one or more AOIs; and determine a state of a surgical activity for the surgical exercise or recognizing the predefined object of interest based on the generated mask being located within a predefined region of interest in the frame of the captured video data. In various embodiments, the video data is captured in real-time.

[0006] In various embodiments, one or more processors of the surgical training evaluator are configured to determine the state of the surgical activity for the surgical exercise based on the generated mask being located within the predefined region of interest in subsequent frames of the captured video data. In various embodiments, one or more processors of the surgical training evaluator are configured to recognize the predefined object of interest based on the generated mask being located the frame of the captured video data. In various embodiments, one or more processors of the surgical training evaluator are configured to track locations of the predefined object of interest based on locations of the generated mask in subsequent frames of the captured video data. In various embodiments, one or more processors of the surgical training evaluator are configured to calculate a metric based on the track locations of the predefined object of interest. In various embodiments, the calculated metrics and / or feedback based on the calculated metrics and / or tracking of objects of interest are provided in real-time, e.g., via a screen or display presented to a user performing or who performed the surgical activity.Docket No.: 10133-USP-PCT

[0007] In various embodiments, the surgical trainer comprises a cover and a base; a camera connected to a lower surface of the cover of the surgical trainer; a light connected to the lower surface of the cover of the surgical trainer; and an exercise insert adapter removably connected to the base of the surgical trainer, the surgical exercise being removably connected to the exercise insert adapter.

[0008] In various embodiments, the surgical exercise comprises at least one of a peg board surgical exercise, precision cutting surgical exercise, tissue dissection surgical exercise, ligating loop surgical exercise, a sleeve transfer surgical exercise or a petal surgical exercise.

[0009] Many of the attendant features of the present invention will be more readily appreciated as the same becomes better understood by reference to the foregoing and following description and considered in connection with the accompanying drawings in which like reference symbols designate like parts throughout.Brief Description of the Drawings

[0010] The present invention may be better understood taken in connection with the accompanying drawings in which the referenced numerals designate like parts throughout the figures thereof.

[0011] FIG. 1 is a perspective view of a surgical trainer and surgical exercises in accordance with various embodiments of the present invention.

[0012] FIG. 2 is a block diagram of a surgical performance automated assessment system in accordance with various embodiments of the present invention.

[0013] FIG. 3 is a perspective view of a surgical trainer and a surgical exercise in accordance with various embodiments of the present invention.

[0014] FIG. 4 is a cross-sectional view of a surgical trainer and a surgical exercise in accordance with various embodiments of the present invention.

[0015] FIG. 5 is a bottom view of a cover of a surgical trainer and a camera in accordance with various embodiments of the present invention.

[0016] FIG. 6 is a perspective view of a camera in accordance with various embodiments of the present invention.

[0017] FIG. 7 is a bottom view of a cover of a surgical trainer and a camera in accordance with various embodiments of the present invention.Docket No.: 10133-USP-PCT

[0018] FIG. 8 is a perspective view of an adapter in accordance with various embodiments of the present invention.

[0019] FIG. 9 is a bottom view of an adapter in accordance with various embodiments of the present invention.

[0020] FIG. 10 is a perspective view of an adapter and a base of a surgical trainer in accordance with various embodiments of the present invention.

[0021] FIG. 11 is a perspective view of an adapter, a surgical exercise, and a base of a surgical trainer in accordance with various embodiments of the present invention.

[0022] FIG. 12 is a perspective view of an adapter, a surgical exercise, and a base of a surgical trainer in accordance with various embodiments of the present invention.

[0023] FIG. 13 is a top view of an adapter, a surgical exercise, and a base of a surgical trainer in accordance with various embodiments of the present invention.

[0024] FIG. 14 is a perspective view of an adapter, a surgical exercise, and a base of a surgical trainer in accordance with various embodiments of the present invention.

[0025] FIG. 15 is a top view of an adapter, a surgical exercise, and a base of a surgical trainer in accordance with various embodiments of the present invention.

[0026] FIG. 16 is a perspective view of an adapter and a surgical exercise in accordance with various embodiments of the present invention.

[0027] FIG. 17 is a bottom view of a surgical exercise in accordance with various embodiments of the present invention.

[0028] FIG. 18 is an exemplary camera view of a surgical exercise, an adapter, and surgical instrument in accordance with various embodiments of the present invention.

[0029] FIG. 19 is an exemplary graphical representation of regions of interest in accordance with various embodiments of the present invention.

[0030] FIG. 20 is an exemplary graphical representation of masks in accordance with various embodiments of the present invention.

[0031] FIG. 21 is an exemplary graphical representation of contours in accordance with various embodiments of the present invention.

[0032] FIGS. 22-24 are exemplary camera views of a surgical exercise, an adapter, and a surgical instrument in accordance with various embodiments of the present invention.Docket No.: 10133-USP-PCT

[0033] FIG. 25 is an exemplary graphical representation of a mask in accordance with various embodiments of the present invention.

[0034] FIG. 26 is an exemplary camera view of a surgical exercise, an adapter, and a hand in accordance with various embodiments of the present invention.

[0035] FIG. 27 is an exemplary graphical representation of a mask in accordance with various embodiments of the present invention.

[0036] FIGS. 28A-B are exemplary camera views of a surgical exercise and an adapter in accordance with various embodiments of the present invention.

[0037] FIG. 28C is exemplary graphical representation of areas of interest for a surgical exercise in accordance with various embodiments of the present invention.

[0038] FIG. 29 is exemplary graphical representation of areas of interest for a surgical exercise in accordance with various embodiments of the present invention.

[0039] FIG. 30 is an exemplary camera view of a surgical exercise and an adapter in accordance with various embodiments of the present invention.

[0040] FIG. 31 is exemplary graphical representation of areas of interest for a surgical exercise in accordance with various embodiments of the present invention.

[0041] FIG. 32 is exemplary graphical representation of areas of interest for a surgical exercise in accordance with various embodiments of the present invention.

[0042] FIGS. 33A-C is a flow diagram of steps or operations performed by a surgical performance automated assessment system in accordance with various embodiments of the present invention.

[0043] FIGS. 35A-56 are exemplary screen views provided by a surgical performance automated assessment system in accordance with various embodiments of the present invention.

[0044] FIG. 57A is a perspective view of a surgical exercise, an adapter, and a base of a surgical trainer in accordance with various embodiments of the present invention.

[0045] FIG. 57B is a top view of a surgical exercise, an adapter, and a base of a surgical trainer in accordance with various embodiments of the present invention.

[0046] FIG. 57C is a perspective view of a surgical exercise and an adapter in accordance with various embodiments of the present invention.

[0047] FIG. 57D is a bottom view of a surgical exercise and an adapter in accordance with various embodiments of the present invention.Docket No.: 10133-USP-PCT

[0048] FIG. 58A is an exemplary camera view of a surgical exercise with exemplary representations of regions of interests in accordance with various embodiments of the present invention.

[0049] FIG. 58B is an exemplary graphical representation of a characteristic pattern for a surgical exercise of FIG. 58A in accordance with various embodiments of the present invention.

[0050] FIG. 58C is an exemplary camera view of a surgical exercise with exemplary representations of regions of interests in accordance with various embodiments of the present invention.

[0051] FIG. 58D is an exemplary graphical representation of a characteristic pattern for a surgical exercise of FIG. 58C in accordance with various embodiments of the present invention.

[0052] FIG. 58E is an exemplary camera view of a surgical exercise with exemplary representations of regions of interests in accordance with various embodiments of the present invention.

[0053] FIG. 60 is an exemplary camera view of a surgical exercise with exemplary representations of contours in accordance with various embodiments of the present invention.

[0054] FIG. 61A is an exemplary camera view of a surgical exercise in accordance with various embodiments of the present invention.

[0055] FIG. 61B is an exemplary representation of inside out version of the surgical exercise of FIG. 61 A in accordance with various embodiments of the present invention.

[0056] FIGS. 62A-65C are exemplary screen views provided by a surgical performance automated assessment system in accordance with various embodiments of the present invention.

[0057] FIG. 66 is a perspective view of a surgical exercise, an adapter, and a base of a surgical trainer in accordance with various embodiments of the present invention.

[0058] FIG. 67A is a top view of a surgical exercise, an adapter, and a base of a surgical trainer in accordance with various embodiments of the present invention.

[0059] FIG. 67B is a perspective view of a surgical exercise and an adapter in accordance with various embodiments of the present invention.

[0060] FIG. 67C is a perspective view of a surgical exercise in accordance with various embodiments of the present invention.

[0061] FIG. 68 is a top view of a surgical exercise in accordance with various embodiments of the present invention.Docket No.: 10133-USP-PCT

[0062] FIGS. 69A-71C are exemplary screen views provided by a surgical performance automated assessment system in accordance with various embodiments of the present invention.

[0063] FIG. 72 is a block diagram of portions of a surgical performance automated assessment system in accordance with various embodiments of the present invention.

[0064] FIG. 73 is an exemplary screen view provided by a surgical performance automated assessment system in accordance with various embodiments of the present invention.

[0065] FIG. 74 is a flow diagram of steps or operations performed by a surgical performance automated assessment system in accordance with various embodiments of the present invention.

[0066] FIGS. 75A-L are exemplary screen views provided by a surgical performance automated assessment system in accordance with various embodiments of the present invention.

[0067] FIGS. 76A-B are top views of a surgical exercise in accordance with various embodiments of the present invention.

[0068] FIGS. 77A-D are exemplary graphical representations of a mask of a portion of a surgical exercise in accordance with various embodiments of the present invention.

[0069] FIGS. 78A-79 are exemplary screen views provided by a surgical performance automated assessment system in accordance with various embodiments of the present invention.

[0070] FIG. 79 is a perspective view of a surgical exercise and a surgical trainer in accordance with various embodiments of the present invention.

[0071] FIG. 80 is a top view of a surgical exercise and a surgical trainer in accordance with various embodiments of the present invention.

[0072] FIG. 81 is a perspective view of a surgical exercise in accordance with various embodiments of the present invention.

[0073] FIGS. 82-86 are exemplary screen views provided by a surgical performance automated assessment system in accordance with various embodiments of the present invention.

[0074] FIGS. 87A-C are exemplary graphical representations of a mask and / or regions of interest in accordance with various embodiments of the present invention.

[0075] FIG. 88 is a top view of a portion of a surgical trainer in accordance with various embodiments of the present invention.

[0076] FIG. 89A-F are exemplary graphical representations of a surgical instrument in relation to a port and a camera in accordance with various embodiments of the present invention.Docket No.: 10133-USP-PCT

[0077] FIG. 90A-E are exemplary screen views provided by a surgical performance automated assessment system in accordance with various embodiments of the present invention.Detailed Description

[0078] In accordance with various embodiments, surgical performance automated assessment systems and / or methods are provided, and various views of various embodiments of the surgical performance automated assessment systems and aspects thereof are shown in the figures. In accordance with various embodiments, the surgical performance automated assessment system is configured to process video data to determine objective metrics and based on these metrics, the surgical performance automated assessment system determines skill assessments and / or surgical proficiency. In various embodiments, the surgical performance automated assessment system is configured to provide real-time processing of the video data and real-time feedback including, for example, performance metrics, skill assessments and / or surgical proficiency. In various embodiments, the surgical performance automated assessment system providing real-time processing is configured to maximize processing of the video data and feedback determination utilizing streamlined processing systems and methods. In various embodiments, video data is provided via a surgical trainer configured to receive surgical instruments and surgical exercises in which a user’s interaction therewith is analyzed to determine metrics, skill assessments and / or surgical proficiency.

[0079] In accordance with various embodiments, a surgical performance automated assessment system comprises a surgical trainer 10, a surgical training evaluator 20, one or more surgical instruments 30, and / or one or more surgical exercises 40. In various embodiments, the surgical trainer 10 is configured to receive one or more surgical exercises and / or one or more surgical instruments. In various embodiments, one or more surgical exercises and / or instruments are removably connected to the surgical trainer. In various embodiments, the surgical trainer is removably connected to the surgical training evaluator. In various embodiments, a cable removably connects the surgical trainer to the surgical training evaluator. In various embodiments, the surgical trainer is wirelessly connected to the surgical trainer.

[0080] In accordance with various embodiments, the surgical trainer 10 comprises a camera 15, one or more lights 16, and / or a fan. The surgical trainer comprises a cover 11, a base 14, and one or more legs 12, 13 connecting the cover 11 to the base 14. In FIG. 1, a surgical trainer 10 inDocket No.: 10133-USP-PCTaccordance with various embodiments is provided. The surgical trainer comprises a cover 11, a base 14, and a plurality of adjustable legs 12, 13. In the illustrated embodiment, the surgical trainer 10 comprises four adjustable legs, a pair of front legs 12a, b and a pair of back legs 13a, b. Each front leg 12a, b comprises a front top leg 121a, b connected on one end to the cover 11 and an opposing end connected to a front bottom leg 122a,b. The front bottom leg 122a,b has an end disposed away from the connection between the front top leg and the front bottom leg which is connected to the base 14.

[0081] In various embodiments, the space, cavity, and / or distance between the cover 11 and base 14 is adjustable via the plurality of legs 12, 13 and / or in various embodiments, the cover 11 is collapsible relative to the base 14 via the plurality of legs 12, 13, easing portability or enhancing or facilitating operation of a surgical exercise. In various embodiments, the front top leg 121a, b is movable relative to the front bottom leg 122a, b. In various embodiments, the front top leg 121a, b is slidable relative to the front bottom leg 122a, b and in various embodiments, the front top leg 121a,b slides along a track on the front bottom leg 122a,b in that the front top leg 121a,b and the cover 11 connected to the front top leg 121a, b is movable towards the front bottom leg 122a,b and the base 14 reducing the distance between the cover 11 and the base 14 and / or the size of a cavity disposed or formed between the cover 11 and the base 14. Similarly, the front top leg 121a, b is movable away from the front bottom leg 122a, b and the base 14, increasing the distance between the cover 11 and the base 14 and / or the size of the cavity disposed or formed between the cover 11 and the base 14.

[0082] In various embodiments, each back leg 13a, b comprises a back top leg 131a,b connected on one end to the cover 11 and an opposing end connected to a back bottom leg 132a, b. The back bottom leg 132a, b has an end disposed away from the connection between the back top leg 13 la,b and the back bottom leg 132a, b and this end is connected to the base 14. The back top leg 13 la,b is movable relative to the back bottom leg 132a, b. In various embodiments, the back top leg 13 la,b is slidable relative to the back bottom leg 132a,b and in various embodiments, the back top leg 131a,b slides along a track on the back bottom leg 132a, b in that the back top leg 132a, b and the cover 11 connected to the back top leg 13 la,b is movable towards the back bottom leg 132a, b and the base 14 reducing the distance between the cover 11 and the base 14 and / or the size of the cavity disposed or formed between the cover 11 and the base 14. Similarly, the back top leg 131a,b is movable away from the back bottom leg 132a, b and the base 14, increasing theDocket No.: 10133-USP-PCTdistance between the cover 11 and the base 14 and / or increasing the size of the cavity disposed or formed between the cover 11 and the base 14.

[0083] In various embodiments, the cover 11 is pivotable or may be tilted relative to the base 14, enhancing or facilitating operation of a surgical exercise. In various embodiments, one or more of the plurality of legs 12, 13 are configured to be adjustable to pivot or tilt the cover 11 relative to the base 14 in that the cover 11 is angled relative to the base 14. In various embodiments, the cover 11 disposed parallel relative to the base 14 is not tilted or at a zero degrees position or condition. In various embodiments, the cover 11 in a fully tilted position or condition is angled in that the front portion of the cover 11 is near the front portion of the base 14, e.g., a distance between the front portion of the cover 11 is minimal or near zero relative to the front portion of the base 14 and the back portion of the cover 11 is distal or positioned away from the back portion of the base 14. In various embodiments, the cover 11 in a fully tilted position or condition is angled at predetermined maximum angle, e.g., about twenty-eight degrees, relative to the base 14. In various embodiments, the cover 11 is configured to movable, pivotable, and / or tiltable relative to the base 14 from a zero degrees position to a fully tilted position and vice versa and in various embodiments, the cover 11 is configured to be positioned or temporarily fixed to one or more intermediate positions between the zero degrees position to the fully tilted position and vice versa.

[0084] In various embodiments, the front top leg 121a, b is movable and / or pivotable relative to the front bottom leg 122a,b and in various embodiments, the front top leg 121a,b pivots about a pivot pin connected to the front top leg 121 a, b and the front bottom leg 122a,b in that a front portion of the cover 11 is pivotable towards or away from a front portion of the base 14 while a back portion of the cover 11 is pivotable away or towards a back portion of the base 14. In various embodiments, the back top leg 131a,b is pivotable relative to cover 11 and the back bottom leg 132a, b is pivotable relative to the base 14. In various embodiments, the back top leg 13 la,b pivots about a pivot pin connected to the cover 11 and the back bottom leg 132a,b pivots about a pivot pin connected to the base 14 in that the front portion of the cover 11 is pivotable towards or away from the front portion of the base 14 while the back portion of the cover 11 is pivotable away or towards the back portion of the base 14.

[0085] In various embodiments, the cover 11 comprises a removable plate 17 configured to be removably inserted into an aperture in the cover 11. The removable plate 17, in various embodiments, comprises a plurality of access ports 18 pre-arranged in the plate. In variousDocket No.: 10133-USP-PCTembodiments, the plurality of access ports are configured to provide an access channel to receive a surgical instrument able to be inserted into and withdrawn out of an access channel of an access port. In various embodiments, one or more of the access ports comprise a seal assembly, e.g., a septum seal, sealing against or contacting an outer surface or periphery of a shaft of the surgical instrument and / or a zero seal, providing a seal or closing the access channel, the access port and / or portions thereof in the absence of a surgical instrument inserted therethrough. In various embodiments, the seal assembly provides haptic feedback to the user as a surgical instrument is inserted, manipulated and withdrawn to facilitate the surgical training of the user and / or to adjust the difficulty of a surgical exercise. In various embodiments, the access ports ensure that surgical instruments are able to be inserted into the surgical trainer at pre-determined locations, i.e., where the access ports are located, and in various embodiments, this facilitates the system in identifying, locating and / or tracking of the surgical instrument and / or adjusting the difficulty of a surgical exercise.

[0086] In accordance with various embodiments, a camera 15 is provided. The camera 15 is configured to view and capture an image of a surgical exercise positioned within the cavity of the surgical trainer. In various embodiments, the camera is positioned above the surgical exercise and / or connected to an internal surface of the cover 11. In various embodiments, the camera 15 is fixed to the cover 11, i.e., is not adjustable or movable, e.g., in an X,Y, or Z axis relative to the cover. The camera 15, in various embodiments, is connected to the cover 11 via a mount 151. In various embodiments, the mount 151 has a bracket 152 configured to be connected to the camera 15. A plurality of arms 153 extend from the backet 152 towards the back or distal end of the cover 11. At an end of each of the arms 153 of the mount 151 are holes 154 configured to assist in locating corresponding holes or bosses in the cover to receive screws or similar fasteners to secure the arms 153 of the mount to the cover 11.

[0087] In various embodiments, the camera is angled or tilted relative to the cover 11, e.g., an upper and / or lower surface of the cover 11. In various embodiments, the cover 11 and / or base 14 are substantially planar. In various embodiments, the camera is angled or tilted relative to the cover 11 and / or base 14, with the cover being in the zero degrees position, the fully tilted position, and / or any position therebetween. In various embodiments, the camera is angled or tilted relative to the cover 11 and / or base 14 to ensure a view or camera image of a surgical exercise includes a depth perspective or view of the surgical exercise and / or to reduce glare or over exposure of theDocket No.: 10133-USP-PCTcamera image. In various embodiments, the camera is angled about seventeen degrees relative to the Y axis.

[0088] In accordance with various embodiments, one or more lights 16 are provided. In various embodiments, the one or more lights face the interior or cavity of the surgical trainer and are arranged to illuminate the interior or cavity of the surgical trainer. In various embodiments, the one or more lights are arranged to face one or more surgical exercises and are arranged to uniformly illuminate the one or more surgical exercise. In various embodiments, the one or more lights, e.g., being fixedly connected to the cover and / or adjacent to the camera, are configured to minimize or not produce glare or overexpose the camera image of illuminated surgical exercise, allowing a user to see and / or the camera to capture details of a surgical exercise. In various embodiments, the one or more lights comprises a plurality of LEDs and in the illustrated embodiment, the one or more lights comprises a first LED strip 161 and a second LED strip 162. The first and second LED strips are positioned on the cover 11. In various embodiments, the first and second LED strips positioned next to the removable plate 17 and / or the aperture in the cover 11 arranged to receive the removable plate 17. In various embodiments, the first and second LED strips positioned near the front or proximal end of the surgical trainer and along two sides of the surgical trainer above the base and / or a surgical exercise uniformly illuminates the surgical exercise, minimizes, or does not produce glare or overexposure of the camera image of illuminated surgical exercise, allowing a user to see and / or the camera to capture details of the surgical exercise. In various embodiments, the one or more lights are spaced from the camera and do not directly face the camera to minimize or avoid glare and / or overexposure of a camera image.

[0089] In accordance with various embodiments, the surgical exercises 40 are configured to train and / or teach a user, e.g., a resident or surgeon, in one or more surgical activities, skills and / or tasks. In various embodiments, the surgical exercises are configured to facilitate the system in identifying, locating and / or tracking of a surgical instrument and / or a surgical performance relative to the interaction with the surgical exercise.

[0090] In accordance with various embodiments, each of the surgical exercises 40 comprises a platform and a task trainer or activity. In various embodiments, the platform is removably connected to an exercise insert adapter 45. In various embodiments, the task trainer comprises one or more portions removably connected to the platform and one or more portions fixedly or permanently connected to the platform. In various embodiments, the surgical exercise comprisesDocket No.: 10133-USP-PCTa peg board surgical exercise, precision cutting surgical exercise, tissue dissection exercise, ligating loop exercise, sleeve transfer exercise, and / or a petal surgical exercise. In various embodiments, the peg board surgical exercise 40a, b comprises an activity comprising a plurality of pegs or posts 41 connected to a platform comprising a base board 42. The activity of the peg board surgical exercise further comprises a plurality of rings removably connected to the plurality of pegs or posts. In various embodiments, each of the plurality of pegs extend in an orthogonal direction relative to the base board. In various embodiments, each of the plurality of rings are movable relative to each of the plurality of rings and the base board. In various embodiments, a ring is movable from one post to another post. In various embodiments, a ring is movable from one end of one post, e.g., a first post, closest to the base board, along the length of the first post and off the open end of the first post away from the base board. In various embodiments, a ring is movable to be positioned over an open end of another post, e.g., a second post, moved along the length of the second post and positioned or left on the base on the end of the first post closest to the base board. Each of the plurality of rings has an opening sized, shaped and / or dimensioned to receive a post therethrough. In various embodiments, each of the plurality of rings share or have a common or equivalent shape, size, dimension, and / or color to each other. In various embodiments, each of the plurality of rings has a different or does not have or share a common or equivalent shape, size, dimension, and / or color compared to the base board. In various embodiments, the plurality of rings comprises a plurality set of rings with each ring of a set having or sharing a common or equivalent shape, size, dimension, and / or color to each other ring of the same set. In other words, in various embodiments, the plurality of rings comprises a first set of rings and a second set of rings with each ring of the first set of rings having or sharing a common or equivalent shape, size, dimension, and / or color to each other and each ring of the second set of rings having or sharing a common or equivalent shape, size, dimension, and / or color to each other. However, the first set of rings does not have or share a common or equivalent shape, size, dimension, and / or color with the second set of rings.

[0091] In various embodiments, the petal surgical exercise comprises a task trainer or activity comprising a central mass 44 with a plurality of petals or arms 45 extending away from the central mass. The central mass and petals are removably connected to a platform comprising a petal base board 46. In various embodiments, a stem 47 extends from the central mass. In various embodiments, a plurality of posts or protrusions extends from the petal base board and extendDocket No.: 10133-USP-PCTthrough corresponding apertures in the central mass, removably securing the central mass to the petal base. In various embodiments, each of the plurality of posts or protrusions extend in an orthogonal direction relative to the petal base and / or are located near a central area of the petal base. In various embodiments, each of the plurality of petals or arms are movable relative to each of other and the petal base. In various embodiments, each of the petals or arms are movable from the petal base to the stem extending from the central mass and in various embodiments, the stem is threaded through an aperture each of the petals or arms.

[0092] In various embodiments, each of the plurality of petals or arms share or have a common or equivalent shape, size, dimension, and / or color to each other. In various embodiments, each of the plurality of petals or arms has a different or does not have or share a common or equivalent shape, size, dimension, and / or color compared to the petal base and / or the plurality of posts or protrusions extending from the petal base. In various embodiments, each of the plurality of petals or arms has a different or does not have or share a common or equivalent shape, size, dimension, and / or color compared to the stem. In various embodiments, the stem has a different or does not have or share a common or equivalent shape, size, dimension, and / or color compared to the petal base, the plurality of posts or protrusions extending from the petal base, and / or the plurality of petals. In various embodiments, one of the plurality of petals is different from the other petals and is configured to ensure or point to a predetermined positioning of the petals relative to the petal base. For example, the different petal indicates a top or first position pointing away from a front or proximal portion of the petal base, the exercise insert adapter and / or the base 14 of the surgical trainer. In various embodiments, one of the plurality of petals has a different shape and / or size relative to the other petals.

[0093] As shown in FIGS. 57A-E and 58, in various embodiments, a precision cutting exercise 40d comprises a task trainer or activity comprising a cutting media 71 and a pattern 72. The pattern is disposed on the cutting media and is visibly distinguishable from the cutting media. In various embodiments, the cutting media is configured to be cut by a surgical scissor, such as a laparoscopic scissor. In various embodiments, the pattern is configured to indicate or illustrate a cut path along which the surgical scissor should cut. In various embodiments, the cutting media is configured to be cut into at least two portions, a first inner portion defined by the pattern and a second outer portion outside the pattern and extending to or towards the edges or ends of the cutting media.Docket No.: 10133-USP-PCT

[0094] In various embodiments, the cutting media is removably connected to a platform. The platform comprises a cut base 73. In various embodiments, the cutting media, e.g., a gauze pad or sheet, is configured to have an outer periphery smaller than an inner and / or outer periphery of the platform, e.g., cut base. One or more clips or connectors 74 are fixedly attached to the cut base. The one or more clips or connectors are configured to releasably attach the cutting media to the cut base. In various embodiments, an upper clip 74a is configured to releasably secure a first or top portion of the cutting media. A first lower clip 74b and a second lower clip 74c, in various embodiments, are configured to releasably secure a lower portion of the cutting media. In various embodiments, the first lower clip is separated from the second lower clip or positioned a predetermined distance away from each other to provide cutting access to the lower portion of the cutting media. In various embodiments, the one or more clips are pivotably or rotatably connected to the cut base. In various embodiments, the cutting media is configured to be cut starting from the lower portion of the cutting media towards the pattern and along the pattern of the cutting media and ultimately cut out a portion of the cutting media as defined or delimited by the pattern.

[0095] In various embodiments, the cutting media is a gauze pad or sheet, and the pattern is one or more circles. In various embodiments, the gauze sheet is arranged to be cut in that the portion delimited by the one or more circles is cut out of the gauze sheet. In various embodiments, the gauze sheet is a four-sided polygon, e.g., a square or rectangle. In various embodiments, the pattern is a circle at about the center of the gauze sheet. In various embodiments, the pattern is a pair of concentric circles with a first circle being disposed within a second circle both sharing a common center. The common center, in various embodiments, is the center of the gauze sheet. In various embodiments, the cut base 73 is removably attached to the exercise insert 50 that is removably attached to the base 14 of a surgical trainer.

[0096] In accordance with various embodiments, an exercise insert adapter 50 is provided. The exercise insert adapter, in various embodiments, is removably attached to the base 14 of the surgical trainer and / or one or more surgical exercises are removably attached to the exercise insert. In various embodiments, the exercise insert adapter is removably secured to the base 14 to prevent movement of the exercise insert adapter relative to the base 14. In various embodiments, the exercise insert adapter comprises front edges 51 and / or back edges 52 configured to engage with respective front edges and / or back edges on the base 14 to ensure proper positioning of the exercise insert adapter onto the base 14 and / or secure the exercise insert adapter, e g., preventing orDocket No.: 10133-USP-PCTrestricting movement, e.g., an X and Y axis movement, relative to the base 14. In various embodiments, the exercise insert adapter and / or the base 14 comprises a hook and loop type interface 53 to removably secure the surgical exercise to the exercise insert adapter and / or preventing or restricting movement, e.g., Z axis movement, of the exercise insert adapter relative to the base 14 of the surgical trainer.

[0097] The exercise insert adapter is configured to connect one or more surgical exercises to the base 14 of the surgical trainer. In various embodiments, each of the surgical exercises comprises a platform removably connected to the exercise insert adapter. In various embodiments, the exercise insert adapter is configured to secure and position a surgical exercise to the base 14 of the surgical trainer to prevent movement, e.g., X and Y axis movement, of portions of the surgical exercise connected to the exercise insert adapter relative to the base 14. In various embodiments, the exercise insert adapter comprises a front flange 55a, a back flange 55b, and side tabs 54a, b configured to engage with the platform of the surgical exercise, e.g., the petal base, cut base, and / or the base board. In various embodiments, the front flange, the back flange, and the side tabs engage with corresponding outer edges of the base board. In various embodiments, the front flange and the back flange engage with corresponding outer edges of the base board, and the side tabs engage with corresponding inner edges or slots of the base board. In various embodiments, the front flange, the back flange, and the side tabs engage with corresponding inner edges or slots of the petal base and / or a cut base. In various embodiments, the exercise insert adapter and / or the surgical exercise platform, e g., the petal base, the cut base, and / or the base board, comprises a hook and loop type interface 56 to removably secure the surgical exercise to the exercise insert adapter and / or preventing or restricting movement, e.g., Z axis movement, of a surgical exercise relative to the exercise insert adapter.

[0098] In various embodiments, the exercise insert adapter is configured to secure and position a surgical exercise to the base 14 to ensure the view or camera image of the surgical exercise is optimal, e.g., able to be analyzed and / or evaluated by the surgical training evaluator.

[0099] In accordance with various embodiments, one or more surgical instruments are provided. The surgical instrument comprises a handle, a shaft, and an end effector. In various embodiments, the shaft comprises a sleeve. In various embodiments, one or more surgical instruments are configured to be insertable through one or more access ports of the surgical trainer.Docket No.: 10133-USP-PCT

[0100] The shaft, sleeve and / or end effector comprises an identifier configured to distinguish a distal end of the sleeve and / or shaft from a proximal end of the end effector. In various embodiments, the shafts and / or sleeves of each of the surgical instruments have or share a common or equivalent shape, size, dimension, and / or color to each other. In various embodiments, the sleeves of each of the surgical instruments do not have or share a common or equivalent shape, size, dimension, and / or color with components or various portions of the one or more surgical exercises and / or the exercise insert adapter.

[0101] In accordance with various embodiments, the surgical training evaluator is configured to analyze video data from the camera using semantic segmentation. In various embodiments, the video data provided by the camera is in a format in which a color for each pixel of an image is assigned a red, green, and blue component (RGB) which is converted to a YUV color space, in which a color for each pixel is assigned a luma or brightness (Y) and color components ((U) blue projection and (V) red projection).

[0102] In various embodiments, the brightness (Y) component is ignored or discarded. In various embodiments, regardless of the brightness, e.g., how dark or light an object appears, e.g., if an object gets darker when a shadow crosses over it, object detection and / or identification of an object within the camera view or image is still performed. Additionally, focusing on only color components (U and V), their hue, increases object detection and / or identification processing speed and reduces overall processing time, bandwidth and / or load of the surgical training evaluator, especially as every pixel of an image is examined. In other words, if the surgical training evaluator used RGB components to identify a color of each pixel, all three values are used. By ignoring the Y component of the YUV components, only two values per pixel are used, increasing the performance and efficiency of the algorithm / evaluator. In various embodiments, the video data provided by the camera is converted into only color components (U and V), thereby increasing the speed of conversion of the video data and reducing the processing time, bandwidth and / or load of the surgical training evaluator.

[0103] From that YUV color space, only a UV color space is utilized. Within that UV color space, areas of interest (AOI) are identified. These AOIs are predefined boundaries or areas within the UV color space in which a set or range of hues of a particular color is identified. As the surgical training evaluator examines each pixel of an image, the UV component of a pixel is compared to the UV space identified in and / or bounded by the AOIs. If there’ s a match, e.g., the UV componentDocketNo.: 10133-USP-PCTof a pixel equals and / or is within a maximum and minimum value or boundary of an AOT, the surgical training evaluator indicates that the pixel is of interest, e.g., containing a potential object of interest. If there is no match, the surgical training evaluator determines that the pixel is not of interest, e.g., not containing a potential object of interest. By examining each pixel of an image in this manner, the surgical training evaluator generates an image mask in which, a UV component of a pixel of interest is designated and / or stored, while a UV component of a pixel not of interest is ignored, e.g., blacked out, assigned a black color. FIG. 19 is an exemplary graphical representation of predefined AOIs within a UV color space. FIG. 20 is an exemplary graphical representation of an image mask generated based on the camera image shown in FIG. 18. The exemplary graphical representations are provided for illustrative purposes to facilitate understanding of the disclosure and are not displayed or other provided to the user.

[0104] In accordance with various embodiments, each AOI represents a corresponding object of interest within an image, e.g., the video data, provided by the camera. For example, AOI 61a corresponds to ring 62a, AOI 61b corresponds to ring 62b, AOI 61c corresponds to tilt indicator 70, and AOI 61 d corresponds to surgical instrument 80. In various embodiments, each AOI represents a set of UV components or values in which a predetermined percentage of these values determines an identification threshold. If an identification threshold is not met or exceeded, the system is configured to determine that an AOI may need to be redefined or adjusted as the current AOI may not be correctly identifying the corresponding object of interest. In such cases, in various embodiments, the surgical training evaluator may generate an error and / or initiate a calibrate or recalibrate operation adjusting the identification threshold to identify an object of interest.

[0105] As shown in FIG. 20, the generated mask represents or identifies the objects of interest within the camera image. For example, ring mask 63a corresponds to ring 62a, ring mask 63b corresponds to ring 62b, tilt mask 63c corresponds to tilt indicator 70, and instrument mask 63d corresponds to surgical instrument 80. The generated image mask, e.g., a digitized version, is used by the surgical training evaluator to determine or infer a state of a current activity and / or to recognize and / or track an object of or utilized with a surgical exercise, e.g., rings, tilt indicators, surgical instruments and the like. In accordance with various embodiments, target objects, objects predefined to be objects of interest, are configured to have distinct colors from each other, allowing for accurate and efficient segmentation of the objects by the surgical training evaluator. Although object identification or semantic segmentation as described, for example, in reference to FIGS. 18-Docket No.: 10133-USP-PCT20, in terms of color, other representations, e.g., two dimensional numerical values or symbols, defining unique and individual AOIs could be utilized.

[0106] Referring to FIG. 21, in various embodiments, utilizing the image mask from segmentation, the surgical training evaluator is configured to determine the contour or outline (i.e., contour determination) of an object mask and / or its corresponding object of interest. In various embodiments, pixels not matching or corresponding to an AOI are blacked or whited out, and thus, starting from a predefined point or origin, a pixel of interest that borders or next to a pixel of interest and a pixel not of interest corresponds to a border or boundary pixel. The surgical training evaluator, by determining and identifying each successive border or boundary pixel, determines an overall contour or outline of the object mask and its corresponding object of interest. A pixel of interest of one AOI that borders a pixel of interest of a different AOI can also represent a border or boundary pixel and further confirmation or determination of the pixel being a border or boundary pixel may depend on the preceding or successive identified border or boundary pixel. In various embodiments, further confirmation or determination of the pixel is deferred to analysis of successive frames of video data or, in various embodiments, ignored to increase processing time and / or reduce processing power.

[0107] As shown in FIG. 21, the generated outline mask represents or identifies the contour or outline of objects of interest within the camera image. For example, outline ring mask 64a corresponds to the contour of ring 62a, outline ring mask 64b corresponds to the contour of the ring 62b, outline tilt mask 64c corresponds to the contour of the tilt indicator 70, and outline instrument mask 64d corresponds to the contour of surgical instrument 80. The generated outline or contour mask, e.g., a digitized version, is used by the surgical training evaluator, in various embodiments, to determine or infer a state of a current activity and / or to recognize and / or track an object of or utilized with a surgical exercise.

[0108] In various embodiments, the surgical training evaluator comprises one or more processors utilizing segmentation and in various embodiments, using segmentation, are configured to analyze video data including being configured to convert pixels from the video data to a YUV color space in which each converted pixel is assigned a brightness component (Y), a first color projection component (U), and a second color projection component (V); identify one or more areas of interest (AOIs) in the converted pixels within a UV color space from the YUV color space, the UV color space comprising the first and second color projection components of the convertedDocket No.: 10133-USP-PCTpixels and ignoring the brightness component, each AOI of the identified one or more AOTs corresponding to a predefined object of interest; generate a mask designating pixels in a frame of the captured video data that correspond to an AOI of the identified one or more AOIs; and determine a state of a surgical activity for the surgical exercise or recognizing the predefined obj ect of interest based on the generated mask being located within a predefined region of interest in the frame of the video data.

[0109] In accordance with various embodiments, the surgical training evaluator is configured to determine if and / or how much the surgical trainer is tilted, e.g., the cover 11 of the surgical trainer tilted or angled relative to the base 14 of the surgical trainer. In various embodiments, two tilt indicators 70 are provided and arranged to remain in the camera view of the camera, even if the cover is tilted or not tilted relative to the base. In the illustrated embodiment, the tilt indicators are circular filled dots having a predetermined color. As the surgical trainer is tilted to different angles, however, the position of the tilt indicators changes in a single or predetermined direction, e.g., vertically as viewed on a display. As such, a lookup table is provided that correlates the vertical height or position of the tilt indicators with an angle of the trainer. For example, as shown in FIG. 23, the tilt indicators at a zero height or lowest position correlates to a maximum tilt or a fully tilted position of the surgical trainer with the tilt indicators at a maximum height or position correlates to a no tilt or zero degrees position as shown in FIG. 22. In accordance with various embodiments, the surgical training evaluator using semantic segmentation identifies and isolates the tilt indicators from the rest of the camera image and contour determination is used to find and determine the location of the tilt indicators, e.g., a pixel or pixel locations of the contour or outline of the tilt indicator relative to the rest of the pixels of the camera image or a predefined image space, e.g., a bottommost corner or edge of a camera image representing an origin or zero pixel location point of the predefined image space. The surgical training evaluator using the determined location, e.g., the pixel locations of the outline of the object within the predefined image space, and the lookup table, e.g., comparing the pixel locations to predetermined pixel locations stored in the lookup table, determines the angle of the surgical trainer. In various embodiments, one or more surgical activities require that the surgical trainer is angled at a specific angle and / or not angled, e.g., at a zero-degree position, the surgical training evaluator utilizing the tilt indicators determines and / or verifies that the surgical trainer is properly angled or not angled as required by the particular surgical activity.Docket No.: 10133-USP-PCT

[0110] In various embodiments, the tilt indicators are positioned at a predetermined location or area relative to the surgical exercise and / or the camera to facilitate the isolation, identification and / or determination of tilt indicators, their position, and relative changes thereto due to the surgical trainer being tilted. In other words, by providing a limited location or area, the surgical training evaluator can be configured to focus on the limited location to isolate and locate the tilt indicators increasing processing time and reducing processing power / load.

[0111] In various embodiments, the exercise insert adapter removably connected the base 14 and a platform of a surgical exercise removably attached to the exercise insert adapter is configured to position the entire surgical exercise within the camera view of the camera fixedly attached to the cover, even if the cover is tilted or not tilted relative to the base. Additionally, in various embodiments, one or more tilt indicators, e.g., two tilt indicators 70, are provided on the exercise insert adapter and / or the surgical exercise to ensure the tilt indicators also remain within the camera view of the camera even if the cover is tilted or not relative to the base. More or less tilt indicators can also be provided with more tilt indicators providing additional accuracy in identifying and determining the tilt of the surgical trainer.

[0112] Surgical Instrument Tracking

[0113] In various embodiments, the surgical training evaluator using semantic segmentation identifies and isolates the surgical instrument 80 from the rest of the camera image and contour determination is used to find and determine the contour or outline the surgical instrument. The surgical training evaluator is configured to identify the two longest sides 81a,b of the surgical instrument outline, as shown in FIG. 25. The surgical training evaluator determines an average line between the two sides, e.g., a line parallel to and midway between the two longest sides, to determine a two-dimensional direction of the surgical instrument and determining the intersection of the average line 82 with the contour of the surgical instrument provides the location of the instrument tip or distal end of the shaft of the surgical instrument. By recognizing and tracking the direction and location of the instrument tip, the surgical evaluator determines or infers a state of a current activity, events happening during the surgical activity, and / or for calculating metrics further utilized to determine or assess a user's skill or performance.

[0114] In accordance with various embodiments, in a two surgical instrument setup or surgical activity, the surgical training evaluator, determining the direction of the surgical instrument is configured to determine or distinguish one surgical instrument from another and / or determiningDocket No.: 10133-USP-PCTand / or designating one surgical instrument to be a left surgical instrument and the other surgical instrument being the right surgical instrument. In various embodiments, the surgical training evaluator, determining the direction of the surgical instrument pointing from the left to the right relative to the surgical trainer and / or display, identifies the surgical instrument to be the left surgical instrument and the other the right surgical instrument. In various embodiments, the surgical training evaluator, determining the direction of the surgical instrument pointing from the right to the left relative to the surgical trainer and / or display, identifies the surgical instrument to be the right surgical instrument and the other the left surgical instrument.

[0115] Hand Detection

[0116] In accordance with various embodiments, a hands data lookup table is provided with a predetermined set or values of UV components that correspond to UV components of thousands of images of people’s hands of a variety of ages, genders, and races under identical lighting and / or other environmental conditions. In various embodiments, as shown in FIGS. 26-27, the surgical training evaluator using semantic segmentation constantly checking or analyzing each pixel of the entire camera image for the presence of a predetermined amount of pixels of a camera image or to determine and count the number of pixels of the camera image that match or correspond to the UV components in the hands data lookup table. In various embodiments, the surgical training evaluator upon determining that a number pixels match the values in the hands data lookup table is larger than a predefined threshold, the system notifies the user that an interference, e.g., a hand 84a and hand mask 84b, was detected, and, in various embodiments, no grade will be provided at the end of the attempt of the surgical activity. In various embodiments, the hands lookup table is limited to ungloved hands or gloved hands having a specific and unique identifier or color in that the surgical training evaluator utilizing semantic segmentation can isolate and identify ungloved or particularly identified gloved hands.

[0117] In accordance with various embodiments, surgical performance automated assessment systems and / or methods are provided, and various views of various embodiments of the surgical performance automated assessment systems and aspects thereof are shown in the figures. In accordance with various embodiments, the surgical performance automated assessment system is configured to process video data to determine objective metrics and based on these metrics, the surgical performance automated assessment system determines skill assessments and / or surgical proficiency. In various embodiments, the surgical performance automated assessment system isDocket No.: 10133-USP-PCTconfigured to provide real-time processing of the video data and real-time feedback including, for example, performance metrics, skill assessments and / or surgical proficiency. In various embodiments, the surgical performance automated assessment system providing real-time processing is configured to maximize processing of the video data and feedback determination utilizing streamlined processing systems and methods. In various embodiments, video data is provided via a surgical trainer configured to receive surgical instruments and surgical exercises in which a user’s interaction therewith is analyzed to determine metrics, skill assessments and / or surgical proficiency.

[0118] In accordance with various embodiments, the surgical training evaluator utilizing semantic segmentation, contour determination, and / or instrument tracking is configured to determine one or more metrics of a user’s performance of a surgical activity. In various embodiments, such metrics include temporal metrics referring to the time-based aspects of a surgical activity. These metrics provide insights into the efficiency and pacing of a user’s, e.g., a surgeon’s, actions. In various embodiments, one temporal metric is total exercise time that measures the duration from the start to the end of the surgical activity. In various embodiments, it is calculated as:where:- Ttotai is the total exercise time,- T start is the start time of the exercise and obtained as the first instance of the surgical tool shows up in the video,- Tend is the end time of the exercise and obtained as the last instance of the surgical tool shows up in the video.

[0119] In various embodiments, the surgical training evaluator is configured to determine motion-based metrics derived from surgical instrument tracking throughout the surgical activity. These metrics, in various embodiments, provide a detailed analysis of the surgeon's movements, offering insights into their precision, efficiency, and dexterity. One such metric is tool tip path length that measures the total distance traveled by the tip or distal end of the shaft of the surgical instrument. It is calculated as the sum of distances between consecutive points along the tool tip path:Docket No.: 10133-USP-PCTwhere:- L is the tool tip path length,- xi and yi are the coordinates of the tool tip at the ith point,- n is the total number of frames tracked.

[0120] In various embodiments, another metric is motion smoothness. Motion smoothness evaluates the fluidity of the surgical instrument movements. It can be quantified using the jerk metric, which is the rate of change of acceleration. A lower jerk value indicates smoother motion:where:- J is the motion smoothness is calculated as the inverse of jerk,- xi and yi are the coordinates of the tool tip at the ith point,- n is the total number of frames tracked.

[0121] In various embodiments, another metric is centeredness or closeness to the center of the video image frame. This metric assesses the precision of surgical instrument movements relative to the center of the video frame. It is calculated as the average distance of tool tip positions from the frame center:where:- D is the total distance from the center,- xi and yi are the coordinates of the tool tip at the ith point,- Xcenter and y center are the coordinates of the frame center,- n is the total number of frames tracked.Docket No.: 10133-USP-PCT

[0122] In various embodiments, another metrics is bimanual dexterity that evaluates the coordination between the surgeon's left and right hands. It is quantified as the ratio of the total path lengths of the left and right surgical instrument tool tips or shaft ends:where:- BD is the bimanual dexterity,- LL is the tool tip path length of the left hand,- LR is the tool tip path length of the right hand.

[0123] Pegboard Exercise

[0124] In accordance with various embodiments, the surgical training evaluator is configured to continuously utilize instrument tracking and hand detection to identify and detect the current state or condition of a surgical activity utilizing a pegboard surgical exercise. In various embodiments, the surgical training evaluator is configured to utilize the tilt indicators, instrument tracking, and hand detection to verify a setup state or condition of a surgical activity utilizing a pegboard surgical exercise.

[0125] In various embodiments, the surgical training evaluator detects the current state of the pegboard and rings. In one embodiment, different pegboards can be used, and the surgical training evaluator is configured to identify which particular pegboard is being used. In various embodiments, the surgical training evaluator analyzes the platform or board base of the pegboard exercises to determine which pegboard is being used. Utilizing semantic segmentation, the surgical training evaluator determines a ratio of white to gray pixels identified in a camera image of the surgical exercise and / or its platform. As shown in the illustrated embodiments, a first pegboard 40b (FIG. 28A) is mostly white, has a mostly white platform, while a second pegboard 40a (FIG. 30) is mostly gray, has a mostly gray platform. As such, the surgical training evaluator identifying the white pixels exceeding gray pixels indicates the first pegboard and gray pixels exceeding white pixels of a camera image of the surgical exercise indicates the second pegboard.

[0126] In various embodiments, if the first pegboard is identified, the surgical training evaluator is configured to identify its orientation relative to the base of the surgical trainer and / or the exercise insert adapter. In various embodiments, the surgical training evaluator is configured to determine if the pegboard is placed either with the rectangular pattern of pegs on the left or theDocket No.: 10133-USP-PCTright. To identify the difference between the two, the surgical training evaluator utilizing semantic segmentation and contour determination isolates and identifies the pegs in the middle of the pegboard and counts the number of pegs. If the surgical training evaluator determines that the pegs in the middle of the pegboard have a pattern with three pegs on the left and two pegs on the right, then the rectangular pattern of the pegs of the pegboard is on the left / the non-rectangular pattern of the pegs is on the right, and vice versa. After identifying the pegboard type and its orientation, the surgical training evaluator sets a plurality of regions of interest (ROIs) 67 overlaying the pegs on the pegboard, matching the peg pattern on display as illustrated in FIG. 28C.

[0127] In various embodiments, as shown in FIG. 29, the surgical training evaluator utilizing semantic segmentation and contour determination isolates and identifies a bottommost peg 168, e.g., a peg closest to the front or proximal portion of the surgical exercise, or lack thereof utilizing one or more regions of interest (ROIs) 167a,b to determine if the pegboard is placed either with the rectangular pattern of pegs on the left or the right. If the surgical training evaluator determines that the bottommost peg is on the left (or absent on the right), then the rectangular pattern of the pegs of the peg board is on the right / the non-rectangular pattern of the pegs is on the left, and vice versa.

[0128] In various embodiments, the different pegboards use different rings, with the second pegboard using yellow rings, while the first pegboard 40b uses orange and green rings 62a, b, as shown, for example, in FIGS. 28A-C. In various embodiments, the surgical training evaluator upon determining the pegboard type, a different semantic segmentation selection or AOIs are used to isolate and identify the rings. In various embodiments, the surgical training evaluator is configured to identify an object mask, e.g., count the number of pixels that match the particular AOI representing a ring, inside the ROI 67 for each peg. If the count is above a predetermined threshold, then that peg is marked or identified as having a ring around it by the surgical training evaluator. In various embodiments, as part of a setup check or condition for a surgical activity utilizing the pegboard, the surgical training evaluator determines that six of the correct or expected rings are present on the board base, and all on the same side. For the activity to be successfully completed, the rings must all pass to the opposite side of the board, then back. In various embodiments, the surgical training evaluator continuously monitors the ROIs 67 of the pegboard to identify and / or mark occupied pegs, i.e., pegs with rings, and unoccupied pegs, to track the progress or states of the activity and / or its successful or unsuccessful completion, e.g., in real-time.Docket No.: 10133-USP-PCTIn various embodiments, a ROI that encompasses the entire surgical exercise may be defined and used by the surgical training evaluator to verify, track and / or determine the setup, progress or states of the activity, and / or other conditions, such as hand detection, removal / addition of surgical instrument, and / or objects, e.g., rings, dropped out of scene.

[0129] Petal Threading

[0130] In accordance with various embodiments, the surgical training evaluator is configured to continuously utilize instrument tracking and hand detection to identify and detect the current state or condition of a surgical activity utilizing a petal threading surgical exercise 40c. In various embodiments, the surgical training evaluator is configured to utilize the tilt indicators, instrument tracking, and hand detection to verify a setup state or condition of a surgical activity utilizing a petal threading surgical exercise.

[0131] In various embodiments, the surgical training evaluator detects the current state of the flower as shown for example in FIGS. 31-32. In various embodiments, the surgical training evaluator isolates and identifies the flower, its central mass 44 and petals 45 from the camera image and designates a plurality of regions of interests (ROIs), e.g., six regions of interest (ROIs) 68 and a seventh ROI 69. The ROIs 68 have fixed sizes and are at fixed locations in the camera image and the corresponding petal threading surgical exercise. In various embodiments, five ROIs are located at the end of each petal 45, having similar sizes, and the sixth ROI is located at the center mass 44.

[0132] In various embodiments, the surgical training evaluator is configured to identify an object mask, e.g., count the number of pixels that match the particular AOI representing a petal and / or center mass, inside the ROIs. If the count is above a predetermined threshold, then that ROI is marked or identified as being active, e.g., having a petal and / or center mass within the ROI by the surgical training evaluator. Otherwise, the ROI is marked or identified as inactive, or unoccupied. In various embodiments, as part of a setup check or condition for a surgical activity utilizing the petal threading surgical exercise, the surgical training evaluator determines that all six ROIs are active. As such, in various embodiments, when setting up the surgical exercise, the center ROI confirms that the flower is placed on the surgical exercise platform correctly, and the other five ROIs confirm that each petal is fully extended and in its correct orientation.

[0133] For the surgical activity to be successfully completed, all five petals must be closed and present in the middle or at the center mass of the flower. In various embodiments, the surgicalDocket No.: 10133-USP-PCTtraining evaluator continuously monitors the ROIs of the petal threading surgical exercise to identify and / or mark active ROIs, i.e., petals present / open, and inactive ROIs, i.e., unoccupied or closed petals, to track the progress or states of the activity and / or its successful or unsuccessful completion. In various embodiments, the surgical training evaluator is configured to identify a successful completion of the surgical exercise by identifying or determining that a seventh ROI 69 that encompasses the five ROIs is inactive or unoccupied by a petal and the sixth or center ROI is active, occupied by one or more petals. In various embodiments, the seventh ROI is an area delimited or defined by concentric circles. In various embodiments, the seventh ROI is an area delimited or defined by an outer circle or ring and an inner circle or ring having a diameter smaller than the diameter of the outer ring. In various embodiments, an ROI that encompasses the entire surgical exercise may be defined and used by the surgical training evaluator to verify, track and / or determine the setup, progress or states of the activity, and / or other conditions, such as hand detection, removal / addition of surgical instrument, and / or objects, e.g., rings, dropped out of scene.

[0134] Precision Cutting

[0135] In accordance with various embodiments, the surgical training evaluator is configured to continuously utilize instrument tracking and hand detection to identify and detect the current state or condition of a surgical activity utilizing a precision cutting surgical exercise 40d. In various embodiments, the surgical training evaluator is configured to utilize the tilt indicators, instrument tracking, and hand detection to verify a setup state or condition of a surgical activity utilizing a precision cutting surgical exercise.

[0136] When setting up the surgical exercise, the surgical training evaluator detects, determines, or identifies that a predetermined gauze or type of gauze is within the camera view. In various embodiments, the predetermined gauze or gauze type comprises a testing gauze 72, a practice gauze 71’, an inner gauze portion or cutout 711 and an outer gauze portion or cutout 712. In various embodiments, the testing gauze 71 includes a single circle or ring 72. In various embodiments, a practice gauze 71’ includes two concentric circles or rings 72’. In various embodiments, the inner gauze cutout 711 comprises a center portion of the testing gauze 71 and the outer gauze cutout 712 comprises a portion of the testing gauze that surrounds the inner gauze cutout 711. In various embodiments, the testing gauze is the inner gauze cutout, and the outer gauze cutout combined. In various embodiments, the surgical training evaluator is configured to distinguish between different gauzes, e.g., the testing gauze, the practice gauze, the inner gauzeDocket No.: 10133-USP-PCTcutout and / or the outer gauze cutout, and / or separately identify each and / or a subset of them. In various embodiments, the inner gauze cutout is the negative space of the outer gauze cutout.

[0137] In various embodiments, the surgical training evaluator identifies or determines what predetermined gauze or gauze type is within the camera view by measuring or determining brightness values of a captured camera image or video. The surgical training evaluator, in various embodiment, determines the brightness values along a line starting from the center of the image / video to a predetermined end point or edge of the image / video.

[0138] The surgical training evaluator, in various embodiments, determines the brightness of each of the pixels within a plurality of regions of interest, e.g., ROI 175, 177, as shown in FIGS.59A-F. In various embodiments, ROI 175 is an area delimited by a circle surrounding the center of the gauze and / or the camera image and having a predetermined radius. In various embodiments, ROI 177 is an area defined by a ring having a width or thickness set or measured between an inner edge or circular periphery and an outer edge or circular periphery.

[0139] In various embodiments, brightness of a pixel is a grey scale value of the pixel based on the red, green, blue (RGB) conversion in which Grey = Red * 0.2126 + Green * 1.7152 + Blue * 0.0722. Thus, each pixel in the converted greyscale image can range from 0 (black, no brightness), to 255 (white, maximum brightness). In various embodiments, the surgical training evaluator performs or executes the RGB conversion to Greyscale utilizing the captured or supplied camera image / video and in various embodiments, the conversion is limited to the captured or supplied camera image / video within the plurality of ROIs. The surgical training evaluator computes or determines an average brightness in each of the ROIs. In various embodiments, the average brightness is determined by summing the greyscale values, e.g., ranging between 0 to 255, of each pixel in a ROI divided by the total number of pixels in that ROI. If the difference between the brightness averages of the ROIs, e.g., ROI 175, 177, is below a constant or predetermined threshold, then the surgical training evaluator determines that the testing or practice gauze is present within the camera view. As such, the system detects or recognizes that the ROIs have or share a common characteristic, e.g., brightness or color, e g., white, that is common or the same between the testing and practice gauzes.

[0140] If the difference between the brightness averages of the ROIs, e.g., ROI 175, 177, is above or exceeds a constant or predetermined threshold, then the surgical training evaluator determines that the inner or outer gauze portion is present within the camera view. As such, theDocket No.: 10133-USP-PCTsystem detects or recognizes that the ROIs have a different characteristic, e.g., brightness or color, (a white gauze covers one region, while a darker grey background covers the other region) between the inner and outer gauze portions. To further distinguish between inner and outer gauze portions, the system detects which region is brightest or whiter, e.g., has a higher greyscale or average greyscale value. If the inner region, e g., ROI 175, is brighter, e.g., has a higher greyscale value or average greyscale value, than the outer region, e.g., ROI 177, then the inner gauze portion is present within the camera view. If the outer region, e.g., ROI 177, is brighter, e.g., has a higher greyscale or average greyscale value, than the inner region, e.g., ROI 175, then the outer gauze portion is present within the camera view.

[0141] To distinguish between the testing gauze versus the practice gauze, i.e., determine or identify if the testing gauze is within the camera view or the practice gauze is within the camera view, the surgical training evaluator, in various embodiments, determines the brightness along a line 176 starting from the center of the image / video to a predetermined end point or edge 178 of the image / video, and in various embodiments, after or once the surgical training evaluator determines that a testing gauze or a practice gauze is within the camera view as previously described. In various embodiments, the determined brightness values along the line provide a predetermined or recognizable pattern that is utilized by the surgical training evaluator to determine the gauze type, e.g., a testing gauze or a practice gauze.

[0142] In the illustrated embodiment, as shown graphically in FIG. 59B, the brightness values as determined by the surgical training evaluator show or indicate a pattern of white, black, and white indicating that a testing gauze 71 is present, e.g., within the camera view. In various embodiments, the brightness values determined by the surgical training evaluator indicate a pattern of a first set of brightness values 171a, a second set of brightness values 171b, and a third set of brightness values 171c, in which the first set and third set of values are the same plus or minus an error margin and the second set of brightness values are different from the first and / or the third set of brightness values to indicate that a testing gauze is present. The surgical training evaluator, by determining or recognizing this pattern determines or identifies that a testing gauze is present.

[0143] In the illustrated embodiment, as shown graphically in FIG. 59D, the brightness values as determined by the surgical training evaluator show or indicate a pattern of white, black, white, black, and white indicating that a practice gauze is present, e.g., within the camera view. In various embodiments, the brightness values determined by the surgical training evaluator indicate a patternDocket No.: 10133-USP-PCTof a first set of brightness values 171a’, a second set of brightness values 171b’, a third set of brightness values 171c’, and a fourth set of brightness values 171 d’, in which the first and fourth set of values are the same plus or minus an error margin and the second and third set of brightness values are same to each other but are different from the first and / or the fourth set of brightness values plus or minus an error margin to indicate that a practice gauze is present. The surgical training evaluator, by determining or recognizing this pattern determines or identifies that a practice gauze is present.

[0144] To determine or identify a gauze type, e.g., to determine or recognize these patterns between the testing gauze and the practice gauze, the surgical training evaluator determines the minimum and maximum brightness values along a line 176 starting from the center of the image / video to a predetermined end point or edge 178 of the image / video. In various embodiment, as the surgical training evaluator determines a brightness value of each pixel along the line, if the brightness value is higher than any previous brightness value, then that brightness value becomes the maximum brightness value, and that process continues until the end of the line is reached. Similarly, in various embodiments, as the surgical training evaluator determines a brightness value of each pixel along the line, if the brightness value is lower than any previous brightness value, then that brightness value becomes the minimum brightness value, and that process continues until the end of the line is reached. With the maximum and minimum brightness values, the surgical training evaluator determines a midpoint or halfway point between the two values. In various embodiments, the surgical training evaluator determines or counts the number of times the determined brightness value crosses the halfway point. If the surgical training evaluator determines that the determined brightness value only crosses, e.g., starts above and then falls below or is greater than and decreases to less than (trends downward or decreases past), the halfway point or threshold once, then the surgical training evaluator determines that a testing gauze is present within the camera view. If the surgical training evaluator determines that the determined brightness value crosses, e.g., starts above and then falls below or is greater than and decreases to less than (trends downward or decreases past), the halfway point or threshold multiple times, e.g., two, then the surgical training evaluator determines that a practice gauze is present within the camera view. In the illustrated embodiments, the top line 170a, 170a’ in the graphs represents the maximum brightness value, line or threshold, the bottom line 170c, 170c’ represents the minimumDocket No.: 10133-USP-PCTbrightness value, line or threshold, and the middle line 170b, 170b’ represents the middle or halfway point, line or threshold between the top and bottom lines.

[0145] In various embodiments, a gauze may be full of small holes and / or the pattern, e.g., ring, may be poorly defined or printed or otherwise distinguishable from the gauze. To filter out noise and inaccurate or poor readings for such a scenario, the brightness value determination for the sampling line is performed by the surgical training evaluator multiple times in multiple directions. In various embodiments, the sampling line is rotated around a whole predetermined circle or outer periphery including the predetermined end point or edge of the image / video. The steps or distance between the sampling lines may vary or be constant. In various embodiments, the steps or number of sampling lines utilized is about 20 to 40 lines or about 30 lines, with the number of samplings increasing accuracy but also increasing processing time and power and vice versa. With each sampling line being moved or rotated, the surgical training evaluator creates brightness value determinations along the new sampling line, e.g., a new graph is created, that is identified or categorized as either of the two gauze types, e.g., a practice or a testing gauze. When a full rotation or predetermined movement of the sampling line is complete, the most identified or categorized gauze type is determined by the surgical training evaluator. The most identified gauze type is then deemed or recognized by the surgical training evaluator as type of gauze within the camera view, e.g., a testing gauze or a practice gauze.

[0146] Accuracy metric

[0147] In various embodiments, the system determines an accuracy metric of the cut performed for the precision cutting exercise. The system, in various embodiments, determines or identifies a perfect accurate cut if no gauze lies or is disposed outside the outer edge or periphery of the ring of an inner gauze portion and no gauze lies or is disposed inside the inner edge or periphery of the ring of an outer gauze portion.

[0148] In various embodiments, the surgical training evaluator determines or identifies any gauze portion disposed outside an outer edge or periphery of the ring of an inner gauze portion by using the average brightness determination within a plurality of regions of interest, e.g., ROI 175, 177, as used in the gauze type identification or determination process. The midpoint between the two averages is used as a cutoff value for the image to remove or subtract any portion of the image below or not above, e.g., less than this brightness cutoff value. As such, in various embodiments, any pixel having a brightness value below the cutoff value is removed, e.g., set to a brightnessDocket No.: 10133-USP-PCTvalue of zero, thereby removing the background and any image noise in the image, leaving only gauze, e.g., any portion of the image above or greater than the brightness cutoff value. The gauze being the brightness or having brightness values above the cutoff value is thus the only thing that remains. Utilizing contouring as previously described the surgical training evaluator outlines the edges of the remaining gauze of the inner gauze portion. The surgical training evaluator identifies or recognizes that the largest contoured portion must be the inner area of the inner gauze portion limited by the ring and is marked or delineated with a circle 185 as shown in FIG. 60. Any delineated contours 186 outside of this circle must be gauze that has been incorrectly cut. In various embodiments, the surgical training evaluator determines or calculates the accuracy metric for the inner gauze portion by adding or summing up all the area of each of the outlined contours outside of the circle. The area of an outlined contour is calculated using the following shoelace triangle formula:where x, y coordinates are vertices or points that form the outlined contour.

[0149] In various embodiments, the surgical training evaluator determines or identifies any gauze portion disposed inside an inner edge or periphery of the ring of an outer gauze portion by using the average brightness determination within a plurality of regions of interest, e.g., ROI 175, 177, as utilized in the gauze type identification or determination process. However, during the surgical activity, one side of the gauze is cut to towards the center of the gauze, causing the outer gauze portion to be split. To remove, reduce or account for this split to properly outline or determine the contour of any gauze within the outer gauze portion, the image of the outer gauze portion is turned inside out. In various embodiments, the surgical training evaluator turns the image inside out by converting the location of each pixel of the image from its Cartesian coordinates to polar coordinates. Polar coordinates have a component rho that represents the distance from the center. By subtracting the rho component or distance from the radius of the image and converting back to Cartesian coordinates, points or pixels close to center get mapped to the edges, while points on the edges get mapped to the center. To remove the initial cut to towards the center in this inside out image, a convex hull 187 is used, as shown in FIG. 6 IB. This convex hull is or represents the smallest convex polygon that contains the contour 188 delimiting the gauzeDocket No.: 10133-USP-PCTas determined by the surgical training evaluator. In various embodiments, the convex hull is determined or calculated by the surgical training evaluator using a Graham scan algorithm or the like. When the convex hull is turned inside out and returned to or applied to the original image, the gap within the outer gauze portion closed or accounted for as shown by the circle 185’ (FIG.61 A). Any delineated or outlined contours 186’ inside of this circle is determined to be gauze that has been incorrectly cut. In various embodiments, the surgical training evaluator determines or calculates the accuracy metric for the outer gauze portion by adding or summing up the area of each of the outlined contours inside of the circle. In various embodiments, the surgical training evaluator determines the accuracy metric for the precision cutting surgical exercise by combing or adding the summed areas of the incorrectly cut portions of the inner gauze portion to the summed areas of the incorrectly cut portions of the outer gauze portion as described above.

[0150] In various embodiments, the lights, fan, and / or camera of the surgical trainer are connected to a hub, e.g., a USB-C hub. The hub is connected to a cable which is connected to the surgical training evaluator. In various embodiments, the surgical training evaluator supplies power to power the lights, fan and / or camera via the cable and hub. In various embodiments, the fan is configured to prevent the camera from overheating. In various embodiments, image data from the camera is supplied to the surgical training evaluator via the cable and hub. In various embodiments, control data or signals configured to modify operations of the lights, fan, camera and / or hub are supplied from the surgical training evaluator to the respective lights, fan, camera, and / or hub. In various embodiments, the hub is connected or integrated into the surgical trainer.

[0151] In various embodiments, the surgical training evaluator is configured to store video data from the camera. In various embodiments, the surgical training evaluator is configured to process in real-time video data from the camera to determine objective metrics applicable to surgical skill assessment. The surgical training evaluator is configured to process one or more of these metrics and to determine and assign pass and fail scores. In various embodiments, the same video data is stored in a database for subsequent post-processing and further analysis.

[0152] In various embodiments, the surgical training evaluator is configured to analyze video data to identify and monitor specific pre-determined structures, surgical instruments, and hand movements. In various embodiments, based on the analyzed video data, the surgical training evaluator is configured to determine, measure, and / or calculate performance metrics such as precision, speed, and accuracy of movements of a user performing a specific surgical activity. InDocket No.: 10133-USP-PCTvarious embodiments, based on the determined, measured and / or calculated performance metrics, the surgical training evaluator is configured to determine skill assessments and / or evaluate surgical proficiency.

[0153] In various embodiments, regions of interest are predetermined based on the surgical exercise and / or surgical activity and in various embodiments, the number and / or size of the regions of interest are predetermined based on the surgical exercise and / or activity. In various embodiments, the surgical trainer is configured to precisely position and secure one or more surgical exercises, one at a time, inside the surgical trainer. In various embodiments, the surgical trainer is configured to provide substantially uniform lighting to illuminate the surgical exercise. In various embodiments, lights of the surgical trainer are configured to illuminate details of the surgical exercise without producing glare or overexposing the camera image. In various embodiments, the colors of the surgical exercises, surgical instruments, and / or other objects or items of interest are configured to facilitate with the semantic segmentation. In various embodiments, the camera is configured to provide a video resolution to supply sufficient data for surgical training evaluator to evaluate surgical performances. In various embodiments, the surgical trainer is removably connected to the surgical training evaluator, easing portability and / or individualized upgradability, customization and / or compatibility of the surgical trainer and / or the surgical training evaluator.

[0154] In various embodiments, the surgical training evaluator comprises one or more processors and / or computing devices configured to receive and analyze video / image data and / or configured to execute machine-readable instructions to analyze video / image data. The surgical training evaluator, in various embodiments, is configured with specialized local applications and / or web browser-based applications configured to receive and analyze video / image data. Exemplary computing devices may include laptops or desktops. In various embodiments, the surgical training evaluator comprises one or more processors and / or computing devices and / or with web browser-based applications that may operate parallelly on multiple processors threads and in various embodiments, may operate in the background so as to not interfere with user operations. In various embodiments, the surgical training evaluator processes and / or the camera provides video data at 60 frames per second, and the surgical training evaluator is configured to provide multiple threaded processing. In various embodiments, the web browser-based applications are written using web-based language (e.g., Web Assembly) such that the applicationsDocket No.: 10133-USP-PCTcan be run off browsers or the cloud / web. In various embodiments, specific application(s) / program(s) are provided or included in the surgical training evaluator to use APIs or other routines to access the camera. Data, along with the access to the camera, is then passed on to web-based application so that it generates and renders corresponding user interface and images that will be displayed on a display for viewing and interactions.

[0155] FIG. 33 (A-C) provides a flow chart (3300) illustrating an exemplary embodiment being performed by the surgical performance automated assessment system and / or one or more processors of the surgical training evaluator. In accordance with various embodiments, the surgical performance automated assessment system provides a selection of surgical activities, and the surgical performance automated assessment system is configured to receive a selection indicator of the selected activity (3301). In various embodiments, this selection can be made on the system via a display screen (e.g., keyboard, mouse) or a touch screen. In various embodiments, the selection of the activity can be from a drop-down menu. In various embodiments, the selection of the activity can be from a list of many compatible activities that can be performed with the surgical trainer. In various embodiments, a display can also be provided in connection with the surgical trainer where selection of the appropriate activity based on the surgical exercise arranged inside the simulated laparoscope can be performed.

[0156] Once an activity is selected or determined by the system to be selected, the system initiates or and / or seeks to establish a connection and start communicating with the camera (3302). In various embodiments, the surgical performance automated assessment system is configured to provide set up instructions via a side panel on the display. The set-up instructions can be detailed with providing what aspects of the surgical activity and / or trainer need to be arranged. In various embodiments, the set-up instructions may be a step-by-step list of written steps. In various embodiments, the set-up instructions may be a video.

[0157] In various embodiments, if the system is not able to connect and communicate with the camera, an error can be provided by the system (3303), via, e.g., on a display. In various embodiments, the system provides an error indicating that the surgical trainer and / or camera is not connected to the surgical training evaluator. In various embodiments, the system requires the camera and / or surgical trainer connection to be resolved, e.g., established, prior to continuing.

[0158] Once the camera has been confirmed to be connected, in various embodiments, a confirmation of such can be shown by the system to the user on a display. This can be shown,Docket No.: 10133-USP-PCTfor example, as an icon that represents that the camera is currently communicating with the system. In various embodiments, the system provides a user selectable “set-up check” operation (3304). The system, in various embodiments, checks the setup of the selected surgical activity (3305). For example, the system checks the setup by comparing video data of the surgical exercise within view of the camera to a plurality of pre-determined setup conditions or thresholds. In various embodiments, different surgical activities have different pre-determined setup conditions. In various embodiments, the surgical activities share one or more common predetermined setup conditions. As shown in FIGS. 34A-B, an example table is provided which provides example surgical activities and their respective setup checks. In the illustrated embodiment, the example surgical activities include Pegboard Performance (3401), Peg Transfer (3402), Petal Threading Performance (3403), and Precision Cutting (3404). It should be noted that additional activities may be provided, and those additional activities may have their own unique respective checks. As seen in the illustrated embodiment shown in FIGS. 34A-B, a common set of checks and pre-determined setup conditions for surgical activities, e.g., Pegboard Performance (3401), Peg Transfer (3402), Petal Threading Performance (3403), and Precision Cutting (3404) are provided. For example, the system checks or determines if the following set of pre-determined conditions are satisfied: the user’s hands are outside of the camera view or no hands are within the camera view, all instruments outside of the camera view or no surgical instruments are within the camera view, the exercise insert adapter and / or the tilt indicator is within the camera view, and / or the cover is in the fully tilted position. In various embodiments, the system determines if the surgical exercise within the camera view matches the expected surgical exercise for the selected surgical activity. Based on the selected surgical activity, the system utilizes different pre-determined setup conditions to determine if the surgical exercise within the camera view is arranged properly or arranged as expected. In various embodiments, for the Pegboard Performance (3401) and Peg Transfer (3402) activities, the system determines if the objects used with the surgical exercise (e.g., rings) are correct and / or are arranged in an appropriate starting arrangement. For the Petal Threading Performance (3403) activity, in various embodiments, the system determines if the surgical exercise is attached and / or arranged flat on the petal base and / or if the surgical exercise is oriented properly, e.g., a triangle shaped petal is pointed in a pre-determined position denoting “up” and / or in a 12 o’clock position. For the Precision Cutting (3404) activity, in various embodiments, the system determines if theDocket No.: 10133-USP-PCTsurgical exercise, e.g., a gauze sheet with a predetermined circle pattern, are correct / expected and / or are arranged in an appropriate starting arrangement.

[0159] If the system identifies or determines a setup check has failed, the system provides an error notification (3306). In various embodiments, an error notification provides instructions on how to potentially correct or fix the setup. Having a pre-determined set-up prior to the start of a surgical activity allows for a “control” state so that the same conditions are established for uniform analysis of user performances in connection with the corresponding surgical activity.

[0160] In one example, with the Pegboard Performance (3401), if one or more of the setup conditions are not satisfied, a corresponding error notice is provided to the user on the display. In various embodiments, the error notice may include instructions as to how to remedy the error. For example, if the system detects the user’s hands and / or instruments, the error notice may indicate that the user’s hands and / or instruments are currently detected, and that the user should remove their hands and / or instruments prior to the start of the surgical activity.

[0161] In another example, the system determines if the arrangement of the surgical exercise is correct or incorrect, such as the improper orientation or that the surgical exercise detected is not supported or expected for the particular surgical activity by, for example, analyzing aspects or portions of the surgical exercise (e.g., a number, shape and / or size of pegs, or posts). In various embodiments, markings may be provided with the surgical exercises including for example, logos, arrows, or other shapes that can be used as an orientation, arrangement, or other indications. In various embodiments, the system will provide an error notice and instructions, for example, to replace (e g., if the “wrong” surgical exercise was used) or rearrange the surgical exercise (e.g., if the surgical exercise was determined to be in an unexpected orientation).

[0162] In various embodiments, the system is configured to detect that the tilt of the surgical trainer is incorrect based on the image captured by the camera. In various embodiments, a tilt indicator is used to detect the surgical trainer’s tilt or angled position and / or determine and / or indicate its correct or incorrect positioning. In various embodiments, if the image of the surgical exercise is determined to be off-center or the entire surgical exercise is not within the image, for example, this can indicate that the camera is not oriented properly as a result of the surgical trainer not being properly tilted. This can be remedied by tilting the surgical trainer and in turn adjusting the positioning of the camera within the surgical trainer. The error notice, in various embodiments, can be used to inform the user to fully tilt the surgical trainer and / or how much toDocket No.: 10133-USP-PCTtilt the surgical trainer. In various embodiments, markings or an indicator may be provided with the surgical trainer that assists and / or informs the user how much to tilt the surgical trainer.

[0163] In various embodiments, the system is configured to detect whether the associated objects (e.g., rings) to be used with the surgical exercise are present and / or arranged in their appropriate starting position. In various embodiments, the camera would capture the image of the surgical exercise and determine whether the requisite objects (e.g., rings) are present and if present whether they are arranged in accordance with a pre-defined starting position. Based on stored information, e.g., a pre-determined setup condition, regarding how the surgical exercise should be set up (e.g., the rings are all situated on one side), if the arrangement is not the same, an error notice will be provided to inform the user that set-up of the surgical exercise and its associated objects (e.g., rings) would need to be adjusted. In various embodiments, an image and / or instructions can be provided to the user to inform how the objects (e.g., rings) should be re-arranged on the surgical exercise before the start of the activity.

[0164] Returning back to FIG. 33 A, if, for example, after the user has followed the instructions in the error notices to fix issues with the setup of the exercise, a “verify setup” selection can be made. In various embodiments, the error notice will include a “verify setup” button / option that will appear on the display alongside any information that would help setup the exercise. When the “verify setup” button is selected, the system re-performs the set-up checks (3305). In this way, the “verify setup” selection behaves the same way as the “setup check” button (3304) but is provided after one or more error notices are provided.

[0165] After the system determines that the setup checks are satisfied or successful, the system may display a “start” button (3307) and, in various embodiments, replaces the previous “set up” button or “verify setup” button. In various embodiments, information about the activity to be assessed may be supplied and displayed by the system when the “start” button is selected; replacing any previous instructions (such as instructions for set up that may have been previously provided to the user).

[0166] Between the time the “start” button is displayed (3307) and when the user selects the “start” button (3309) {see FIG. 33B), the system is configured to periodically check (e.g., via the camera) whether the set up for the exercise is still correct. For example, the system is configured to determine if the user’s hands or surgical instruments are not within the camera view until the surgical activity is started. Much like in the previous set-up checks, the system, utilizing videoDocket No.: 10133-USP-PCTdata provided by the camera of the surgical exercise and its surrounding environment, can determine if the set-up remains correct. In various embodiments, the system will ensure that the objects (e.g., rings) were not prematurely moved prior to the start of the activity. If any conditions which were previously checked (3305) are detected, the system is configured to display an error notice or banner (3308) which informs the user that the surgical exercise / environment has been altered and to make changes to fix the setup for the selected surgical activity. If the error banner or notice is displayed (3308), in various embodiments, the system can be configured to disable the “start” button until the error(s) has been addressed.

[0167] When selected by the user, the “start” button can indicate to the system that the exercise / simulated surgical procedure will begin shortly (3309). In various embodiments, detected activation or selection of the “start” button causes the system to initiate recording of the video data provided by the camera and, in various embodiments, the recording can be saved in memory, locally and / or remotely. Activation of the “start” button also indicates when tracking of a user performance should begin. Thus, anything after when the “start” button is pressed, the system detects, measures, quantifies, and / or determines the user’s performance (i.e., assessment period). In some embodiments, a short delay can be provided to allow user to get situated prior to the starting of the assessment period.

[0168] In various embodiments, a text prompt can be used to indicate to the user when to start the exercise after the “start” button has been selected in (3309). In various embodiments, the text prompt may be immediate. In various embodiments, a delay can be implemented prior to the text prompt appearing to allow preparation prior to the start of the exercise. In various embodiments, the text prompt may instruct the user to begin. In various embodiments, the text prompt may provide instructions as to what steps would need to be performed, for example, allowing the insertion of the appropriate instruments into the surgical trainer (3310).

[0169] Once the surgical activity has been started, the system continually monitors the related activities associated with the surgical activity (3311) within the surgical trainer. In various embodiments, the exercise may be broken into different segments / portions with corresponding instructions that would inform what tasks would need to be performed. In various embodiments, hints / suggestions can also be provided to facilitate performance of the activity. The instructions are displayed on a display so reference to the instructions can be performed as needed.Docket No.: 10133-USP-PCT

[0170] During the performance of the activity (3311), the surgical performance automated assessment system is configured to monitor a plurality of different conditions. In various embodiments, three conditions correspond to different conditions that could potentially cause issues with evaluating the performance of the activity. A first condition is having the system detecting one of the user’s hands within the camera view (3312). By using computer vision, in various embodiments, the system is able to detect images that appear to be similar to hands within the environment. In various embodiments, the activity may require only the use of the instrument to manipulate the surgical exercise and / or objects associated with the exercise. To simulate actual surgical procedures, for example, the use of hands is prohibited. Thus, detection of the user’s hands may immediately terminate the surgical activity and cause the system to determine that the activity has not been completed successfully. In various embodiments, warnings may be provided a pre-determined number of times prior to termination of the activity.

[0171] A second condition is having the system failing to detect interaction with the surgical exercise, e.g., the system does not detect surgical instruments or movements of portions of the surgical exercise for a pre-determined period of time (3313). If the system does not detect interaction with the surgical exercise, the system can automatically terminate the recording and / or the surgical activity after not detecting any activity or actions within the surgical trainer for a pre-determined period of time. In various embodiments, a banner can be provided on the display which provides notification that an extended period of non-activity has been detected, e.g., due to not being able to detect surgical instruments within the surgical trainer. The banner may appear after a pre-determined period of time has elapsed after the system has not detected any instruments. The timer used to determine when the banner should appear may be reset every time an instrument is detected. Once the banner appears, a second pre-determined period of time may be used before the surgical activity is terminated. In various embodiments, the banner may provide the second pre-determined period of time as a countdown notifying when the surgical activity will terminate. If any instrument is detected during this period of time, the banner can be removed, and the countdown removed or reset. However, if the second pre-determined period of time elapses, the system will automatically terminate the activity and determine whether the surgical activity was a failure or incomplete.

[0172] A third condition is having the system detect that the current activity is exceeding a pre-determined time period (3314). To ensure completion of an activity within an allotted time,Docket No.: 10133-USP-PCTan overall period of time is provided for each activity; whereby each different activity may be allotted the same or different amounts of time. Furthermore, complexity of the activity may also affect the amount of time being allotted for completion of the activity. The set period of time corresponds to how long the system will be allowed to record the performance of the activity. Thus, when the amount of data being recorded via video by the camera is nearing the maximum length of time allowed, this is indicative of nearing the allowed allotment of time to complete the activity.

[0173] In various embodiments, an example maximum allotted period of time allowed for an activity may be set at 30 minutes. Each activity, which may have their own respective maximum time, will be provided prior to the start of the activity so that the time constraints associated with the activity being performed are known. This time constraint may be provided, in various embodiments, as a countdown timer somewhere on the display. In various embodiments, the countdown may be maintained and known only by the system until it is nearing the end of the countdown. For example, the countdown timer may not become visible until, for example, 5 minutes remains; in which case the countdown timer may become visible, and a notice be provided that the time to complete the activity is nearing. Further thresholds (e.g., 2 minutes, 1 minutes, 30 seconds) may also be used which provide further notice regarding the impending ending of the activity. If after the countdown timer reaches 0 (i.e., the maximum allotted period of time has elapsed), the activity is terminated, and the system concludes the current performance is incomplete or a failure.

[0174] As noted above, when either the maximum time allotted (3314) has been exceeded or the system fails to detect activity (e.g., surgical instruments) for a period of time, the system terminates the current activity. In various embodiments, the system can still provide feedback (3316) up to the point that the activity was performed for, for example, quantifying the performance via one or more graphs. In addition, the system can also provide the various criteria necessary for passing the current activity on the display with the feedback (3316). The criteria may also include notes as to why the current activity failed.

[0175] In various embodiments, in connection with the feedback (3316) being provided with the early terminated activity, an option may be provided on the display which allows the ability to “retry” or “reattempt” the current activity. The surgical performance automated assessmentDocket No.: 10133-USP-PCTsystem may also allow an exiting of the activity completely or the ability to choose a different activity to practice on.

[0176] A fourth condition (3315) is also detectable by the system but does not correspond to potential error or incomplete conditions described above. Rather, the fourth condition (3315) corresponds to the selection via a “finish” or “complete” button. The fourth condition (3315), when selected, indicates that the activity has been completed or the ending of the recording for the present attempt on the activity.

[0177] After the “finish” button has been selected (3315), the system monitors the environment to check to see if all instruments have been removed from the trainer. If one or more instruments are still identified from the image captured by the camera, a notice requesting the removal of the instruments is provided (3317)

[0178] Once all instruments have been removed, the surgical performance automated assessment system begins uploading the video recording of the performance of the recent activity (3318). While the video recording is being uploaded, user feedback is provided. In various embodiments, user feedback includes providing the criteria associated with the activity, scores needed to pass for each of the criteria, current scores, and high scores from past attempts. In various embodiments, graphs can also be provided to provide an illustrative summary of user performance over multiple attempts for one or more of the criteria. Once the upload of the video recording is complete, options are provided on the display to allow a user to “try again” (i.e., repeat the activity that was just completed) or “exit” (i.e., leave the activity).

[0179] With reference to FIG. 35 A, an exemplary display 3500 is shown after the setup of the surgical exercise has been checked by the system ((operation 3305) as illustrated in FIG. 33). Specifically, the exemplary display corresponds to after the setup of the surgical exercise within the surgical trainer and determination that the setup is correct via the notification “setup check complete” 3508.

[0180] As shown, the display is split into two portions. A first portion (on the right) 3501 contains information about the exercise (e g., Pegboard Performance). A second portion (on the left) 3507 displays the real-time image capture of the environment (e.g., the surgical exercise 3509 within the surgical trainer).

[0181] With the first portion (on the right) 3501, the information included on the side includes various information about the exercise that will be performed. The first portion 3501Docket No.: 10133-USP-PCTincludes information such as the name 3502 of the activity (e.g., Pegboard Performance), a [maximum] time duration 3503 for the activity (e.g., 5 minutes), and a number of criteria 3504 that would provide a passing score. The criteria 3504 lists the minimum threshold that would allow for a passing score to be awarded by the surgical performance automated assessment system. Example criteria include how long the activity was performed for and quantifying characteristics of the performance within the activity such as bimanual dexterity, smoothness, and motion efficiency. In addition, the first portion 3501 includes a list of activity instructions 3505 that can detail what cannot be done during the activity lest an invalid attempt be determined. As illustrated in the figure, such instructions may include informing that a hand needs to be kept out of the environment, that the instruments should be maintained within the environment, that the performance of the activity should not exceed a pre-determined period of time, and / or that the activity has not been properly finished. In various embodiments, the list of activity instructions can also include instructions on how to perform the activity properly from the beginning to the appropriate end state.

[0182] In various embodiments, a start button 3506 is provided. When selected, the system starts the selected activity as previously described. The “start” 3506 button may be greyed out or not selectable until confirmation that the instructions and related information for the activity have been read.

[0183] With reference to the second portion (on the left) 3507 of the display, the real-time image of the environment associated with the activity to be performed (e.g., pegboard performance) is shown. The real-time image is acquired by the camera of the surgical trainer. In various embodiments, the real-time image displays at least a portion of the surgical exercise within a framed view that allows adequate viewing of the surgical exercise on the display.

[0184] As shown in the figure, the example real-time image shows the surgical exercise (e.g., pegboard) 3509 situated in the center of the framed visual space. Furthermore, the surgical exercise (e.g., pegboard) 3509 has a plurality of posts or pegs 3512 and corresponding objects (eg., rings) 3510 used with the posts 3512 of the surgical exercise 309. A marker 3512, such as a logo, may also be included with the surgical exercise 3509. The marker 3512 can be used to help orient the surgical exercise 3509 in a right-side-up orientation with respect to the camera (and in turn facilitate proper setup for the exercise using the surgical exercise 3509).Docket No.: 10133-USP-PCT

[0185] Tn various embodiments, a full screen option 3513 can be provided. When selected, the full screen option 3513 may minimize or hide the entirety of the first portion 3501. This would allow expansion of the second portion 3507 so that the real-time image of the environment can be expanded and shown over a larger area. Another option (not shown) can also be provided so that the user is capable of bringing back the first portion as needed; for example, if reference to the activity instructions 3505 need to be performed at a later time during the activity.

[0186] As shown in FIG. 35B, after the “start” button 3506 has been selected, a notification 3514 can be provided on the display informing a user to start the activity. In various embodiments, the notification may include instructions to insert the instrument(s) to be used with the exercise into the surgical trainer. As noted above, the system needs to be able to detect the instruments being used else the exercise be determined to be invalid / failed attempt. In various embodiments, the notification 3514 may include information about what instruments should be used.

[0187] Along with the notification 3514 that would be used to direct the user to start the activity, there is also a “finish” button 3522. The “finish” button 3522 is selectable after completion of all the steps in the activity. In various embodiments, the “finish” button 3522 may be greyed out until the computing system detects all the objects nearing an end state (e.g., all the pegs on an opposite side from a starting point). In various embodiments, the “finish” button 3522 may be greyed out for a pre-determined period of time.

[0188] When selected, the “finish” button 3522 informs the system to stop recording of the performance of the activity. In various embodiments, a prompt may appear that warns the participant that all instruments should be removed from the environment and to ensure that the activity has been fully completed after which another selection of the “finish” button 3522 would terminate the activity. In various embodiments, termination of the activity may automatically be performed a pre-determined period of time after selection of the “finish” button 3522.

[0189] With reference to FIG. 35C, an example illustration is shown regarding additional details that can be shown related to each of the criteria 3504 used to quantify a performance of the activity. Next to each of the criteria 3504 may include an icon 3515 that is selectable. The icon 3515 is tied to additional details 3516 about the corresponding criteria 3504. As seen in the figure, the icon 3515 next to the “motion efficiency” criteria would provide details 3516 such asDocket No.: 10133-USP-PCTwhat “motion efficiency” covers and how the criteria are calculated for purposes of the passing score.

[0190] With reference to FIG. 35D, an example invalid attempt notice 3517 is provided. The notice 3517 can be made immediately distinguishable with markings 3521 that can be immediately discernable that the current attempt has been terminated, or the attempt is determined a failure. In various embodiments, the notice 3517 provides reasons 3518 for the notice such as “the current session timed out due to inactivity.” In an example embodiment, the example notice 3517 and associated reason 3518 may correspond to the scenario that the surgical performance automated assessment system failed to detect user involvement with the activity for an extended period of time. As noted above, the system not being able to detect the instruments for a pre-determined period of time ends the current attempt as the system determines that the user is not participating with the activity. Furthermore, the notice 3517 provides options to restart (i.e., retry) 3519 the current activity or exit 3520 the activity altogether.

[0191] With reference to FIG. 36A, an example display 3600 is shown, similar to the display from FIG. 35 A; provided after a setup check performed by the system. This is illustrated via the notification “setup check completed” 3608. Much like the display in FIG. 35A, the display in FIG. 36A is also split into two portions. The first portion (on the right) 3601 contains information about the activity (e.g., Petal Threading Performance). The second portion (on the left) 3607 displays the real-time image capture of the environment (e.g., the surgical exercise 3609 within the surgical trainer).

[0192] With the first portion (on the right) 3601, the information included on the right side includes various information about the activity that will be performed. The first portion 3601 includes information such as the name 3602 of the activity (e.g., Petal Threading Performance), a [maximum] time duration 3603 for the activity (e g., 10 minutes), and a number of criteria 3604 that would provide a passing score. The criteria 3604 lists the minimum threshold that would allow for a passing score to be awarded by the surgical performance automated assessment system. Example criteria include how long the activity was performed for and quantifying characteristics of the performance within the activity such as bimanual dexterity, smoothness, and centeredness. In addition, the first portion 3601 includes a list of activity instructions 3605 that can detail what cannot be done during the activity lest an invalid attempt be determined. As illustrated, such instructions may include informing that a hand needs to be kept out of theDocket No.: 10133-USP-PCTenvironment, that the instruments should be maintained within the environment, that the performance of the activity should not exceed a pre-determined period of time, and / or that the activity has not been properly finished. In various embodiments, the list of activity instructions can also include instructions on how to perform the activity properly from the beginning to an appropriate end state.

[0193] In various embodiments, a start button 3606 is provided. When selected, the system starts the selected activity as previously noted above. The “start” 3606 button may be greyed out or not selectable until confirmation that the instructions and related information for the activity have been read.

[0194] With reference to the second portion (on the left) 3607 of the display, the real-time image of the environment associated with the activity to be performed (e.g., petal threading performance) is shown. The real-time image is acquired by the camera of the surgical trainer. In various embodiments, the real-time image displays at least a portion of the surgical exercise within a framed view that allows adequate viewing of the surgical exercise on the display.

[0195] As shown, the example real-time image shows the surgical exercise (e.g., petal threading) 3609 situated in the center of the framed visual space. The exemplary surgical exercise 3609 has a post 3612 and with a number of petals 3610 surrounding the post 3612. In various embodiments, a marker, such as a logo, may also be included with the surgical exercise 3609. The marker 3612 can be used to help orient the surgical exercise 3609 in a right-side-up orientation with respect to the camera. In various embodiments, the petals 3610 can help assist in the orientation of the surgical exercise 3609.

[0196] In various embodiments, a full screen option 3613 can be provided. When selected, the full screen option 3613 may minimize or hide the entirety of the first portion 3601. This would allow expansion of the second portion 3607 so that the real-time image of the environment can be expanded and shown over a larger area. Another option (not shown) can also be provided so that the user is capable of bringing back the first portion as needed; for example, if reference to the activity instructions 3605 need to be performed at a later time during the activity.

[0197] As seen in FIG. 36B, after the “start” button 3606 has been selected, a notification 3614 can be provided on the display informing a user to start the activity. In various embodiments, the notification may include instructions to insert the instrument(s) to be used withDocket No.: 10133-USP-PCTthe exercise into the surgical trainer. As noted above, in various embodiments, the system needs to be able to detect the instruments being used else the exercise be determined to be invalid / failed attempt. In various embodiments, the notification 3614 may include information about what instruments should be used.

[0198] Along with the notification 3614 that would be used to direct the user to start the exercise, there is a “finish” button 3622. The “finish” button 3622 is selectable after completion of all the steps in the activity. In various embodiments, the “finish” button 3622 may be greyed out until the system detects all the objects nearing an end state (e.g., all the pegs on an opposite side from a starting point). In various embodiments, the “finish” button 3622 may be greyed out for a pre-determined period of time.

[0199] When the “finish” button 3622 is selected, the system stops recording of the video data from the camera. In various embodiments, a prompt may appear that warns the participant that all instruments should be removed from the environment and to ensure that the activity has been fully completed after which another selection of the “finish” button 3622 would terminate the activity. In various embodiments, termination of the activity may automatically be performed a pre-determined period of time after selection of the “finish” button 3622.

[0200] With reference to FIG. 36C, an example illustration is shown regarding additional details that can be shown related to each of the criteria 3604 used to quantify a performance of the activity. Next to each of the criteria 3604 may include an icon 3615 that is selectable. The icon 3615 is tied to additional details 3616 about the corresponding criteria 3604. As seen in the figure, the icon 3615 next to the “centeredness” criteria would provide details 3616 such as what “motion efficiency” covers and how the criteria are calculated for purposes of the passing score.

[0201] With reference to FIG. 37A, an example feedback screen 3700 is shown. The feedback screen is provided after the ‘finish’ selection is selected (see 3315; FIG. 33) during a pegboard performance activity (see 3318; FIG. 33).

[0202] In the example feedback screen 3700, an upload status bar 3701 is shown which provides a status update on how far along the process of uploading the recording of the video data has been completed. The upload status bar 3701 also provides a warning that informs that the video is being uploaded and that the current page should not be closed. In various embodiments, a numerical value (e.g., percentage) is provided to inform how far along the upload is at. Furthermore, the upload bar can fill up (from left to right) to correspond with theDocket No.: 10133-USP-PCTnumerical value shown. In various embodiments, the numerical value may be a countdown timer which estimates how long until the upload has been completed.

[0203] In various embodiments, while the upload is being performed, information about the performance of the activity is provided. In various embodiments, the feedback is user specific. In the feedback screen 3700, a portion shows the criteria for passing the activity being performed. The criteria are all listed within a criteria box 3702 which includes threshold values for one or more individual criteria that would be considered passing. The criteria are specific to the activity; different criteria may be used for different activities. For example, as shown in the figure, example criteria include time, level of bimanual dexterity, level of smoothness, and level of motion efficiency as tied to the pegboard performance activity. In various embodiments, each of the criteria listed within the criteria box 3702 would need to be at or better the passing level for the activity to have been deemed passing.

[0204] The feedback screen 3700 also includes a current score box 3703. The current score box 3703 includes the current performance and corresponding valuations / scores of the different criteria. The respective scores shown in the current score box 3703 are provided by the system after an initial evaluation of the performance by the user. In the example, each and every one of the criteria has met or performed at a better level than the passing score threshold listed in the criteria box 3702.

[0205] The scores highlighted in the current score box 3703 are calculated based on information determined by the surgical training evaluator regarding each of the criteria specific for the surgical activity. In various embodiments, the surgical training evaluator determines metrics for the surgical activity. In various embodiments, the determined metrics comprise bimanual dexterity, smoothness, motion efficiency or tool tip path length, and / or centeredness. In various embodiments, the surgical training evaluator assigns the determined metric (e.g., bimanual dexterity, smoothness, tool tip path length, centeredness) to a score (e.g., bimanual dexterity score, smoothness score, motion efficiency score, centeredness score). In various embodiments, the surgical training evaluator assigns the determined metric (e.g., bimanual dexterity, smoothness, tool tip path length, centeredness) to a score (e.g., bimanual dexterity score, smoothness score, motion efficiency score, centeredness score) using a percentage or predetermined scale, e.g., 0 to 100. In various embodiments, a high score value, e.g., near 100, is associated with better proficiency with that criterion for the activity. In various embodiments, aDocket No.: 10133-USP-PCTpassing score and / or passing threshold is predetermined, e.g., 80. In various embodiments, to determine a score, for example, shown in the current score box 3703, a predetermined relationship or association between a metric value and the score are used. In various embodiments, the predetermined relationship comprises an equation, where the score = m * (metric) + b; where m and b are dependent on criteria and conditions for passing the criteria.

[0206] As an example, in accordance with various embodiments, one of the criteria determined by the system for the pegboard performance is bimanual dexterity (BD). In various embodiments, a passing score or threshold (e.g., 80) is assigned to a passing BD metric where there is a 10% difference in distance between the left and right tool and a best BD metric value corresponds to 0% difference and is assigned a score of 100. Based on a line formed between the passing score (i.e. .1,80) and best score (i.e. 0, 100), a corresponding slope = -200 and intercept = 100 is obtained. Thus, in various embodiments, the scores for BD can be determined by the surgical training evaluator using the equation (score = -200 * (metric) + 100. Example calculations are provided below:a. If a BD metric is .2 (or 20%), then the corresponding score is -> -200 *.2 + 100 = 60 b. If a BD metric is .1 (or 10%), then the corresponding score is-200 *.l + 100 = 80;which corresponds to a “passing” score or thresholdc. If a BD metric is .05 (or 5%), then the corresponding score is -> -200 *.05 + 100 = 90 d. If a BD metric is 0 (or 0%), then the corresponding score is-200 * 0 + 100 = 100

[0207] Other scores ((e.g., smoothness score, motion efficiency score, centeredness score), for other respective metrics (e.g., smoothness, tool tip path length, centeredness), in various embodiments, can be similarly determined and displayed within the feedback screen.Furthermore, in various embodiments, the scores are used to calculate a high score shown, for example, in the high score box 3704.

[0208] The feedback screen 3700, in the illustrated embodiment, also includes a high score box 3704. The high score box 3704 shows the one attempt with the highest “total” score. To facilitate the “total” score comparison, criteria (such as time) may need to be assigned a value on the 0-100 score value. The best score would be obtained by comparing the “total” scores from all past attempts and selecting the one performance with the highest “total” value. With the example of the pegboard performance exercise, the scores for time, bimanual dexterity, smoothness, and motion efficiency would be combined, e.g., added all together, to provide a totalDocket No.: 10133-USP-PCTthat is compared to totals of other attempts to determine the best attempt with the highest score. In various embodiments, each of the criteria may be provided a specific weight to facilitate in the determination of highest “total” value.

[0209] In various embodiments, the high score box 3704 is user-specific. In various embodiments the high score box 3704 can be universal (i.e., covers multiple users).

[0210] The feedback screen 3700 also includes graph 3705 which illustrates performance of one or more criteria over a number of different attempts. For example, the figure illustrates the graph 3705 showing the difference in time needed to complete the activity by the user over three attempts. Other criteria can also be shown graphically. In various embodiments, the system is capable of displaying each of the criteria based on user selection from the feedback screen 3700. In various embodiments, each of the criteria may be displayed in graphs 3705 altogether (e.g., one on top of each other).

[0211] In graph 3705, each attempt will have a point 3706 on the graph which can be used to comparatively quantify user performance over the number of attempts. In various embodiments, the graph 3705 may also include a threshold line 3707 (denoted by a dotted horizontal line) that corresponds with the passing score for the criteria. The graph 3705 may also include a notification 3708 informing the user of how well the user performed at the current attempt compared to the previous attempt.

[0212] In various embodiments, the graph 3705 may also include a portion associated with suggestions on how to improve 3709 in connection with one or more criteria. In various embodiments, the suggestions 3709 may be specific to the criteria being shown in the graph 3705. In various embodiments, the suggestions 3709 may be specific to the criteria(s) which are not measured as passing. In various embodiments, the suggestions 3709 may be generalized directions on how to improve one or more of the criteria.

[0213] Once the upload bar 3701 has completely filled, the upload bar 3701 may be replaced with an upload completion notice 3710 informing that the upload has been completed within the feedback window 3700. As seen in FIG. 37B, the feedback window 3700 also includes a retry button 3711 and an exit button 3712 near the upload completion notice 3710. The retry button 3711 resets the current activity to the beginning. Setup for the activity would need to be performed (and the system confirms the setup is correct) before the activity can be repeated. The exit button 3712 terminates the current activity and allows selection of a different activity.Docket No.: 10133-USP-PCT

[0214] FIG. 38 illustrates an exemplary feedback screen 3800 which is similar to the feedback screens 3700 shown in FIG. 37A and FIG. 37B except now has graphs 3806-3809 for all the different criteria associated with the activity that was just completed.

[0215] The feedback screen 3800 has the same sections as the previous feedback screen 3700 from FIG. 37A; has the upload status bar 3801, the criteria box 3803, the current score box 3804, and the high score box 3805. Specific to this embodiment, the current performance highlighted in the current score box 3804 is a new high score. This is provided via a notification 3802 so that the user knows that the current performance is the best performance so far. Though the current performance is a new high score, the high score box 3805 has not yet been updated to reflect the new high score. Rather, the high score box 3805 reflects the performance that was the best prior to the current attempt. On the next attempt, the high score box 3805 will be updated to include the scores that were presented in the current score box 3804.

[0216] As mentioned above, various embodiments of the feedback screen can include multiple graphics illustrating user performance for criteria over multiple attempts. In the figure, the feedback screen 3800 as shown in the figure includes four graphs 3806-3809, one graph each corresponding to a different criterion. The first graph 3806 highlights total time scores, e.g., how fast the activity has been completed, the second graph 3807 highlights bimanual dexterity scores, the third graph 3808 highlights smoothness scores, and the fourth graph 3808 highlights motion efficiency scores.

[0217] In various embodiments, each of the graphs may also include suggestions or hints 3810 where applicable to improve the scores with their respective criteria. In various embodiments, the suggestions or hints may only be provided at certain pre-determined thresholds with the aim to improve performance to a next threshold. The suggestions or hints are retrieved from a stored set of suggestions or hints and can be retrieved based on the feedback scores.

[0218] In various embodiments, suggestions or hints for one or more criteria may be static and automatically provided to the user on the feedback screen when their current score is failing. In various embodiments, suggestions or hints may be specific to the user. For example, if a bimanual dexterity score is deemed below the passing threshold, a determination as to which hand was moving more or which hand is the dominant hand (e.g., left vs right) can be made in order to provide corresponding suggestions or hints.Docket No.: 10133-USP-PCT

[0219] FIG. 39A shows an example feedback screen 3900 where a non-passing score has been determined. The feedback screen 3900, in various embodiments, includes the same information as previous feedback screens shown above in FIG. 37A, 37B, and 38. As seen in the figure, the feedback screen 3900 includes the upload status bar 3901, the criteria box 3903, the current score box 3904, the high score box 3905, and the graph 3906. However, this time, the scores in the current score box 3904 do not all meet or exceed the predetermined threshold necessary for passing as highlighted in the criteria box 3903. In various embodiments, the system is configured to make these determinations and / or comparisons and provide the corresponding display. As shown, for example, the bimanual dexterity score in the current score box 3904 is below the necessary passing score identified in the criteria box 3903. As such, the current attempt is determined to be a failure and a “try again” notice 3902 is provided to inform the user that the current attempt is non-passing.

[0220] The feedback screen 3900 also provides an overview of user performance over the various attempts. In various embodiments, the information about user performance for one or more criteria can be shown via one or more graphs 3906, 3907. In various embodiments, a specific graph for a criteria can be chosen to be displayed. In various embodiments, each of the graphs can be shown simultaneously. The graph 3906 can provide details on the score for a particular attempt as well as the date and time that attempt was completed. In various embodiments, suggestions (not shown) can also be provided for each criteria or for specific criteria e.g., non-passing scores). In various embodiments, the suggestions can highlight specific information that would improve performance associated with criteria with non-passing scores so that a passing score may be achievable in the next attempt. Once the upload has been completed, as indicated by the upload status bar 3901, the notice 3902 can change from “try again” to inform that the upload has been completed. Furthermore, “try again” and “exit” buttons can be also provided to allow the ability to start another attempt exit the activity.

[0221] With reference to FIG. 39B, in various embodiments, the feedback screen can be made to display more than one graph (which each graph corresponding to the scores for different criteria being evaluated) instead of just mainly one criteria (as illustrated in FIG. 39A). The transition from the feedback screen as shown in FIG. 39A to the feedback screen as shown in FIG. 39B is possible via selection of more than one criteria the user would like to view. In various embodiments, the feedback screen can be made to automatically display the graph for theDocket No.: 10133-USP-PCTtime-related score for the activity. To display different or additional criteria related graphs, selection of the criteria can be obtained for the graphs to be displayed. In various embodiments, selected criteria can replace the existing graph with the graph of the selected criteria.

[0222] In various embodiments, the feedback screen can be extended and / or the graphs compressed in order to display two or more graphs. As seen in FIG. 39A, one graph is shown fully while other graphs are potentially partially shown. In various embodiments, the size of the feedback screen can be made to extend in a vertical direction thereby allowing the display of multiple graphs simultaneously. The ability to extend the feedback screen in a vertical direction can also be based on the device (e.g., desktop display versus mobile display) being used and / or the dimensions of the display. In various embodiments, re-sizing the information being displayed within the feedback screen can also allow multiple graphs to be displayed simultaneously.

[0223] FIG. 40A illustrates an embodiment of a feedback screen 4000. The feedback screen 4000 is provided after finishing a petal threading activity. In various embodiments, different criteria are provided in the criteria box 4003, while the remaining sections for the feedback screen 4000 are similar to the feedback screen 3700 provided in FIG. 37A. For example, the feedback screen 4000 includes the upload status bar 4001 which provides a current progress on the video uploaded. Furthermore, the feedback screen 4000 includes the criteria box 4003 which includes all the criteria specific for the petal threading activity along with the passing thresholds needed for each criteria; the current score box 4004 which highlight the scores of the current attempt; the high score box 4005 which highlights the best attempt of all the past attempts by the user; one or more graphs of the criteria related to the activity; and suggestions that may be useful in helping the user improve performance in one or more of the criteria.

[0224] In this embodiment of the feedback screen 4000, there is a notice 4002 which informs the user that the current attempt performing the activity is considered a failure. By comparing the threshold score values from the criteria box 4003 and the scores highlighted in the current score box 4004, it can be seen that the attempt failed due to not having a passing bimanual dexterity score.

[0225] Once the upload of the video has been completed, the notice 4008 within the feedback screen 4000 can change to inform the user accordingly. The feedback screen can alsoDocket No.: 10133-USP-PCTinclude a ‘retry’ button 4009 that can be selected so that the same activity can be attempted again and an ‘exit’ button 4010 so that the current activity can be terminated. This, for example, would allow selection of a different activity.

[0226] FIG. 40A-FIG. 40B illustrate exemplary embodiments where an error may occur and are shown on the feedback screen. Specifically, the error arises with the broken communication between the system where the user is performing the activity and where information can be stored and retrieved about each attempt associated with the user. For example, FIG. 41A shows the similar feedback screen 4100 discussed above with sections such as the notice 4101 used to inform the status of the upload of the video, the options to retry and exit the current activity via the retry 4102 and exit 4103 buttons. The criteria box 4104, current score box 4105, and high score box 4106 are also provided. In various embodiments, some information may be missing within the criteria box 4104 and the high score box 4106 due to system errors or connection issues. In various embodiments, the surgical performance automated assessment system is configured to retrieve the associated information about the activity as well as information about the user performing the activity from memory; where such memory can be stored in a separate location from the system where the user is performing the activity. For example, the separate storage may be in the cloud or on a storage device separate from the system. The broken communication (e.g., wi-fi / wired connection being disconnected) between the system and where the information needs to be retrieved from causes the inability to populate the criteria box 4104 and high score box 4106. Furthermore, a broken connection may also prevent the display of graphs and corresponding suggestions to improve performance since those sets of information may also be retrieved from a remote location from the system.

[0227] In various embodiments, some of the information (such as activity specific or userspecific high scores and past performances) may be stored locally on the system as well. This could minimize the impact of broken communication scenarios with external storage locations (e.g., cloud).

[0228] With reference to FIG. 4 IB, retrieval of the activity and user-specific information may not be impacted, for example, if stored locally on the system. However, the notice 4108 in the feedback screen shows an issue with the uploading process (e.g., “video upload as failed”). The failed upload process can be caused by, for example, by issues with recording the video during the performance of the activity or dropped communication between the storage locationDocket No.: 10133-USP-PCTand the system at some point during uploading. If this is the case, in various embodiments, the current attempt is not counted as an attempt for purposes of comparing user performance over time.

[0229] In various embodiments, prior to or simultaneous with the upload to the storage location, the video recording can be saved locally in memory. This would allow for scenarios where the uploading process has failed to allow the user to re-attempt uploading the video at a later time once the communication between the system and the storage location has been fixed. In various embodiments, another page or menu can be provided which would allow the user the ability to select the locally stored video recording, highlight the particular attempt the video recording is associated with, and initiate a manual upload action.

[0230] With reference to FIG. 41C, the feedback screen shows an error 4110 associated with the graph being displayed. In various embodiments, the graph is generated from past performance data of a user where such data is stored in a separate location from the system. Thus, whenever a graph is requested (e.g., selection to view a graph for a particular criteria), past performance data is retrieved and then processed by the system to generate the graph that is displayed in the feedback screen. The notice provided in the error 4110 highlights the possible cause (e.g., the inability to communicate with the storage location over the internet). By following the suggestion to refresh the page, another attempt to retrieve the needed past performance data can be performed to generate the graph. In various embodiments, the past performance data can also be stored locally on the system which would circumvent this possibility.

[0231] FIG. 42A illustrates an embodiment whereby a current attempt was not completed. In particular, the feedback screen 4200 provides a notification 4210 that the activity time limit exceeded, e.g., 5 minutes. As noted above, each activity may have a pre-determined maximum time allotted to complete the activity. In various embodiments, the maximum time allotted is to control the size of the video that would need to be recorded and transmitted.

[0232] When the maximum time allotted for a particular activity has been exceeded, the attempt automatically ends and the feedback screen 4200 is provided. In various embodiments, any recorded video obtained during the time period is uploaded so that a record and be provided as to why the maximum time was reached. In various embodiments, there may be a distinction made / determined by the system with the associated attempt, e.g., if the user performing theDocket No.: 10133-USP-PCTactivity failed to complete the activity within the maximum allotted time period, how far along the user was in the activity, and / or if the maximum allotted time period was met because of a lack of progress (or inactivity). Since the activity was not completed, the current score box 4220 is left empty. In various embodiments, the current score box 4220 could provide a score up to the point in progress to the activity with a marking that the attempt was not complete.

[0233] Even with non-completed exercises, in various embodiments, the attempt is uploaded and stored in memory. Thus, the system, in various embodiments, indicates that the feedback screen 4200 should not be closed until the upload is complete via the notification 4230. Once the upload is complete, the notification 4230 changes to inform the user and then allows the user a choice between retrying the same activity or terminating the activity (and potentially allowing selection of a different activity) 4240 as seen in FIG. 42B.

[0234] FIG. 43A illustrates another exemplary scenario where the activity being attempted may be determined to be incomplete. In connection with the pegboard performance, for example, one of the requirements is to move the rings from one side of the surgical exercise to the opposite side prior to pressing the ‘finish’ button. However, if after the ‘finish’ button was pressed and the system determines that the requirement for the activity has not been fulfilled, a determination that the current attempt as being invalid can be provided. In various embodiments, the determination that the current state of the activity (e.g., the surgical exercise and the associated rings / pegs) does not correspond with the required end state of the exercise is based on the system identifying where the objects (e g., rings) are currently detected and where they should be located.

[0235] The incomplete conclusion is conveyed to the user via the notification 4310 by the system on the feedback screen 4300. Again, because the current attempt was not determined to be a valid attempt, the current score box 4320 is left blank. Furthermore, in various embodiments, the video will be uploaded so that the attempt can be stored, e.g., for future reference. The notification regarding the upload 4330 is provided to the user.

[0236] FIG. 43B illustrates a follow-up feedback screen whereby the upload notification 4340 is provided to confirm that the upload of the video has been completed. The feedback screen provides the user with the options to retry 4350 the activity or to exit 4360 the activity completely (and allow the user to select a different activity if desired).Docket No.: 10133-USP-PCT

[0237] FIG. 44A illustrates a scenario where the activity being attempted may be determined to be incomplete. In particular, with the petal threading performance, one of the requirements is to have every one of the petals threaded onto the stem prior to pressing the “finish” button. However, if after the “finish” button was pressed and the system determines that the requirement for the activity has not been fulfilled, a determination that the current attempt as being invalid can be provided. In various embodiments, the determination that the current state of the activity (e.g., the stem and the petals) does not correspond with the required end state of the activity is based on identifying whether each petal is threaded onto the stem.

[0238] Thus, as shown, for example, the system has determined that the exercise was incomplete due to two petals not being threaded onto the stem. The reason for the incomplete determination is provided via a notification 4420 to the user. The feedback screen 4400 includes the criteria box 4430 which includes the requisite passing scores for each of the criteria being evaluated during the petal threading activity. The high score box 4450 includes the best (i.e., highest scores from a past attempt related to the activity. However, the current score box 1240 is kept blank because the current attempt was not finished / completed. Furthermore, a notice that the video is being uploaded 4410 is provided to the user. Even if the activity failed or otherwise incomplete, in various embodiments, performance video is still uploaded to the storage location.

[0239] As shown in the figure, for example, no graph is provided with the feedback screen 4400. This may be due to the incomplete status of the current attempt of the activity. In various embodiments, the system only prepares a graph for display in the feedback screen 4400 for attempts that completed the activity.

[0240] FIG. 44B illustrates a follow-up feedback screen whereby the upload notification 4460 is provided to confirm that the upload of the video has been completed. The feedback screen provides the user with the options to retry 4470 the activity or to exit 4480 the activity completely (and allow the user to select a different activity if desired).

[0241] FIG. 45 A illustrates another scenario of a failed attempt. In particular, during the pegboard performance, the system detected that a number of objects (e.g., rings) were dropped out of scene during the activity. This may come about when, during the course of performing the activity, the user attempts to move the object (e.g., ring) from one side of the surgical exercise to the opposite side and drops the object in a manner that causes the object to leave the scene (e.g., the view of the surgical exercise captured by the camera within the surgical trainer and / orDocket No.: 10133-USP-PCTdisplayed on the display screen). Tn various embodiments, any objects that leave the scene would cause the end state of the surgical exercise to not match the expected end state of the activity if properly completed. In various embodiments, a pre-determined number of objects may be allowed to leave the scene before the activity is determined to have been a failure.

[0242] As shown, the feedback screen 4500 provides notice that the current activity is a failed attempt 4510. In particular the notice 4510 provides the reasoning that three object(s) were dropped out of the scene. In various embodiments, the system is configured to track how many objects are supposed to be within the scene and thus able to determine how many objects are missing.

[0243] In various embodiments, the system is configured to track the path of the objects as they are moved from one side of the surgical exercise to the opposite side; if any of those objects have a path that intersects the edges of the scene, this can also be indicative that the object has been dropped and left the scene. In various embodiments, detecting such instances may increment a counter to track how any object paths have been detected in a similar manner.

[0244] As shown for example, because the attempt was a failure / incomplete, the current score box 4520 for the attempt is left blank. Otherwise, all other aspects of the feedback screen 4500 are similar to various embodiments already discussed above.

[0245] FIG. 45B illustrates a follow up feedback screen 4500 after the video of the incomplete / failed attempt (illustrated in FIG. 45A) has been uploaded. The notice 4530 informing the user is generated by the system and shown in the feedback screen 4500.Afterwards, the system provides choices to “retry” the activity or “exit” the activity, similar to what was discussed above in the various embodiments.

[0246] With reference to FIG. 46A, another error condition is presented with the feedback screen 4600. In various embodiments, the feedback screen 4600 provides another notice 4610 informing the user that a hand has been detected within the scene with respect to the petal threading performance activity (though this notice is similarly applicable to other activities as well) by the system. In various embodiments, detection of one or both hands in the scene can invalidate an attempt on an activity. In various embodiments, detection of one or both hands beyond a pre-determined period of time can invalidate an attempt of an activity. Such conditions would prevent user interaction with anything other than the surgical instruments which typically best simulates actual surgical procedures.Docket No.: 10133-USP-PCT

[0247] Tn various embodiments, like previous error conditions that invalidate an attempt on the activity being performed, if the system detects one or more of the user’s hands, no scores are recorded for the current attempt. This is shown in the current score box 4620 where no scores are currently presented. Despite not finishing the activity, in various embodiments, the feedback screen will still notify the user that that the video associated with the attempt of the activity is being uploaded 4630. After the video has been uploaded (see FIG. 46B), the corresponding notice is provided to the user. The feedback screen 4600 also provide the choices to “retry” the activity” or “exit” as well.

[0248] FIG. 47 shows another exemplary error condition that can be seen on the feedback screen. In various embodiments, the feedback screen 4700 provides a notice 4710 that indicates that the user’s instruments were not detected for an extended period of time as determined by the system. In various embodiments, the period of time is based on the activity being performed. As illustrated, the period of time was two minutes for the petal threading performance activity. Because the error was detected, in various embodiments, the notice 4710 further indicates to the user that the scores were not provided and / or saved, which corresponds to the current score box 4720 being left blank. Despite the activity being incomplete, in various embodiments, the video associated with the attempt is still uploaded and the upload status bar 4730 provides the notice to the user of the upload progress. After the upload is complete, the feedback screen 4700 notifies the user and allows the user to “retry” or “exit” the activity.

[0249] In various embodiments, a timer notification 4810 can be provided as seen in FIG. 48. As mentioned above, the different activities that can be performed using the surgical performance automated assessment system have a pre-determined maximum time limit that is allotted before the system deems the attempt invalid / failed. As seen in the display screen 4800, the timer notice 4810 provides information to the user that the maximum activity time is almost reached; which for the pegboard performance is identified as being 30 minutes 4820. In various embodiments, the maximum activity time can be adjusted to be more or less based on the exercise and / or the difficulty. In various embodiments, the timer notice 4810 may only appear at pre-determined thresholds (e.g., 15 min, 10 min, 5 min, 2 min, 1 min, etc....) and disappear until another threshold is crossed. In various embodiments, the timer notice 4810 may appear at a first pre-determined threshold and remain on the screen acting as a countdown timer indicating to the user how much time is remaining for the activity. Once the maximum activity time allotted forDocket No.: 10133-USP-PCTthe activity has been reached as determined by the system, the current attempt on the activity will be determined to be incomplete.

[0250] FIG. 49A illustrates an exemplary assessment history page 4900. The assessment history page 4900 provides activity specific information for a specific activity. As shown, the assessment history page 4900 is specific to the pegboard performance activity. From the assessment history page 4900, the surgical performance automated assessment system provides information about a user’s performance related to an activity 4909 and the capability to start the activity 4910. The pegboard performance activity 4909 information includes scores related to the highest or best performance 4911 by the user. In various embodiments, there may also be additional notes 4912 that are directed towards improving performance. In various embodiments, the additional notes 4912 may be specific to the user and / or based on past attempt scores. In various embodiments, the additional notes 4912 may be generic suggestions on how to improve one or more criteria scores. With the highest or best performance 4911 by the user, the surgical performance automated assessment system also provides the associated video recording 4913, allowing, for example, a past performance tied to their best performance on the activity to be viewed. In various embodiments, any past video recording associated with any past attempt can be retrieved and displayed within the assessment history page 4900.

[0251] In various embodiments, a graph 4914 is also provided that shows user attempts over a number of different attempts. The graph 4914 shows one graph for a criteria at a time; with different graphs being displayable based on selecting one of the criteria options 4915 within the graph 4914. As shown, the current graph provides the bimanual dexterity score based on the selection in the criteria options 4915. Furthermore, the graph 4914 shows a progression towards the passing score 4916, which in various embodiments is denoted by a dashed line. In various embodiments, the graph 4914 displays a pre-determined number of previous attempts (e.g., last 10 attempts as illustrated in the figure). In various embodiments, any subsequent attempt would remove the oldest attempt from the graph 4914.

[0252] With reference to the assessment history page 4900, information about whose information is tied to the assessment history page 4900 is shown 4908. Thus, the information shown is user-specific and, in various embodiments, would require a login to access. The assessment history page 4900 also provides the ability to go to other menus associated with other features or functions of the surgical performance automated assessment system. “Home” 4901Docket No.: 10133-USP-PCTbrings back up the main page. “Courses” 4902 brings up the different activities and / or difficulties that can be selected. “My videos” 4903 links or provides past recorded and uploaded video data of past attempts on all the activities associated with the surgical performance automated assessment system. In various embodiments, “Video Library” 4904 provides access to all past recorded and uploaded video data of the other users. “Assessment History” (as discussed above) 4905 provides information about past performances and comparison between the past performances. In various embodiments, “Resources” 4907 may include informational videos that can be viewed to teach how to perform an activity and / or skills associated with the activity.

[0253] FIG. 49B illustrates an exemplary assessment history page 4900. In various embodiments, the assessment history page is arranged in a different orientation compared to FIG.49A to accommodate differing display screens that can be used to view the information. In particular, the more vertical arrangement may be more useful for implementations tied to mobile devices or displays of different dimensions. In various embodiments, the vertical orientation provides the display of multiple graphs simultaneously.

[0254] FIG. 50 illustrates another exemplary assessment history page 5000. As shown, a selection of the bimanual dexterity criteria 5010 and its corresponding graph of the most recent (i.e., last 3) attempts are provided by the system. In various embodiments, the graph includes completion date information 5020 such as when the attempt for the activity was completed, at what time the activity was completed, and how long the activity took to complete. A passing threshold marker 5030 which identifies the threshold that would need to be satisfied to “pass” the criteria is also provided. As shown, the passing threshold marker 5030 corresponds to the passing score of 80. Attempt #3, for example, has a score that is above the passing threshold marker 5030.

[0255] As shown, a zero score is identified for attempts #1 and #2 (5040). In various embodiments, all attempts (even failed / incomplete) are recorded and uploaded into memory. In various embodiments, a 0 score would indicate that no scores were stored for the first two attempts (e.g., attempts 1 and 2). In various embodiments, the system may provide a zero score to indicate that an attempt was incomplete or failed. In various embodiments, information related to why each attempt failed as determined by the system can be provided by the system,Docket No.: 10133-USP-PCTsuch as via a summary that can be temporarily displayed, for example, when a cursor hovers over the point in the graph associated with a failed attempt.

[0256] FIG. 51 illustrates another exemplary assessment history page 5100. In this figure, the system attempts to process and show a recorded video 5110 associated with the attempt having the highest score 5120 being shown. The figure shows the recorded video 5110 is currently being processed, as information about the attempt is retrieved from a stored memory location and configured for display on the display. In various embodiments, the surgical performance automated assessment system would need to format the recorded video to properly be displayed on the display.

[0257] Beneath the recorded video 5110 is a media control feature 5130 which receives input regarding playing and pausing 5150 the recorded video. Details about the length of the recorded video 5140 are provided in the media control feature 5130. The surgical performance automated assessment system further provides a notice of what time the current video is at and allows jumping to various points between the start and end of the recorded video 5110.

[0258] FIG. 52 illustrates another exemplary assessment history page 5200. As shown, the graph 5210 displays details about the user’s bimanual dexterity score over their first three attempts. As shown, the graph has a passing threshold 5230 along with the associated score that would need to be met in order to be determined as a passing score. As illustrated, the third attempt 5220 crosses the passing threshold 5230. The first two attempts 5240 are below the passing threshold 5230 which would indicate a non-passing attempt. In various embodiments, the first two attempts 2040 being at zero may be more indicative of an incomplete or failed attempt.

[0259] FIG. 53 illustrates another exemplary assessment history page 5300. As shown, the highest score / best performance 5310 is provided for each of the criteria associated with the activity being performed (which in this illustrated embodiment is the pegboard performance). The assessment history page 5300, in various embodiments, also shows the performance related to the metric over a number of attempts via a graph 5320. As shown, the graph 5320 includes a threshold marker 5330 indicating where the passing score is located. Furthermore, each point on the graph 5320 indicates the user score for that attempt. For example, in the illustrated embodiment, the first two attempts were likely invalid because the calculated scores were zero. However, starting with attempt 3, the user is shown to have completed each subsequent attemptDocket No.: 10133-USP-PCTsince a non-zero score is shown on the graph 5320. In various embodiments, the graph 5320 provides identification 5340 of the high score and when (e.g., date and time) the attempt was completed. In various embodiments, the identification 5340 can be provided for each of the data points on the graph 5330 upon interaction with that point. For example, if user interaction is detected (e.g., via touch screen, cursor) with the data point of the 6thattempt 5350 in the graph, a corresponding notification can pop up near the data point for the 6thattempt 5350 which provides the score for the criteria and / or the date and time when that attempt was completed.

[0260] Furthermore, as shown for example, there is no video upload associated with the highest score 5340. For this attempt, since no video was uploaded, there is no ability to view this specific attempt in the future. In various embodiments, the failure of the video to upload may be caused by the dropped communication between the system and the stored memory location where the video is supposed to be uploaded. In various embodiments, the failure of the video to upload may also be caused by the termination of the application (e.g., closing of the feedback screen), e.g., prior to the upload complete notice being provided to the user after completion of the attempt.

[0261] FIG. 54 illustrates another exemplary assessment history page 5400. As shown, a video recording 5410 is currently being played and / or displayed by the system. In various embodiments, the video recording can be associated with the displayed highest score performance being currently shown via the assessment history page 5400. In various embodiments, the video recording 5410 that is played within the assessment history page 5400 can be selected from any of the previous completed attempts. Such selection can be made by interacting with one of the data points 5440 shown on the graph 5430 as determined by the system. The video recording 5410, in various embodiments, has an associated media control feature 5420 which provides information about the length of the video recording, where the user is currently at when viewing the video recording, the ability to play, stop, pause the video recording, and the ability to jump around to different points in the video recording.

[0262] FIG. 55A and FIG. 55B illustrate exemplary assessment history pages 5500. The exemplary assessment history pages 5500, in various embodiments, are initially provided prior to any attempts being performed by the user in connection with the pegboard performance (FIG. 55A) and the petal threading performance (FIG. 55B). As shown, there are no high scores to post and thus no graphical data to present based on past performance. As soon as one attempt isDocket No.: 10133-USP-PCTcompleted, the assessment history pages 5500 can populate the highest score information and corresponding graphs (as discussed above).

[0263] FIG. 56 illustrates another exemplary assessment history page 5600 associated with a Peg Transfer activity. Like the above discussed assessment history pages for the peg performance and petal threading activities, the assessment history page 5600 for the Peg Transfer activity has sections to provide related user information. As shown, the assessment history page 5600 includes the highest score section 5610 that provides the scores for the best performance / attempt on the activity; including the date and time the attempt was completed as well as the associated scores for each of the criteria (e.g., time, bimanual dexterity, smoothness, motion efficiency). If the best scores are each passing, the high score section 5610 will also denote that the attempt is passing. A video recording 5620 may also be provided alongside the high scores. In various embodiments, the video recording 5620 may be the video recording of the attempt. In various embodiments, the video recording may be instructions for the activity. The assessment history page 5600, in various embodiments, also includes the graph portion 5630 which illustrates data of one or more of the metrics over a pre-determined number of attempts (e g., 10). The graph 5630 also includes a passing threshold 5640 that can indicate what score is needed for passing.

[0264] With reference to FIG. 62A, an exemplary display is shown after the setup of the surgical exercise has been checked by the system ((operation 3305) as illustrated in FIG. 33). In various embodiments, the exemplary display corresponds to after the setup of the surgical exercise within the surgical trainer and determination that the setup is correct via the notification “setup check complete” 3508.

[0265] As shown, the display is split into two portions. A first portion (on the right) 3501 contains information about the exercise (e g., Pegboard Performance). A second portion (on the left) 3507 displays the real-time image capture of the environment (e.g., the surgical exercise 3509 within the surgical trainer).

[0266] With the first portion (on the right) 3501, the information included on the side includes various information about the exercise that will be performed. The first portion 3501 includes information such as the name 3502 of the activity (e.g., Precision Cutting), a [maximum] time duration 3503 for the activity (e.g., 6 minutes), and criteria 3504 indicating a passing score. The criteria 3504, in various embodiments, lists a minimum threshold that wouldDocket No.: 10133-USP-PCTallow for a passing score to be awarded by the surgical performance automated assessment system. Example criteria include how long the activity was performed for and quantifying characteristics of the performance within the activity such as smoothness and accuracy. In addition, the first portion 3501 includes a list of activity instructions 3505 that can detail what cannot be done during the activity lest an invalid attempt be determined. As illustrated in the figure, such instructions may include informing that a hand needs to be kept out of the environment, that the instruments should be maintained within the environment, that the performance of the activity should not exceed a pre-determined period of time, and / or that the activity has not been properly finished. In various embodiments, the list of activity instructions can also include instructions on how to perform the activity properly from the beginning to the appropriate end state.

[0267] In various embodiments, a start button 3506 is provided. When selected, the system starts the selected activity as previously described. The “start” 3506 button may be greyed out or not selectable until confirmation that the instructions and related information for the activity have been read.

[0268] With reference to the second portion (on the left) 3507 of the display, the real-time image of the environment associated with the activity to be performed (e.g., precision cutting) is shown. The real-time image is acquired by the camera of the surgical trainer. In various embodiments, the real-time image displays at least a portion of the surgical exercise within a framed view that allows adequate viewing of the surgical exercise on the display.

[0269] In various embodiments, a full screen option 3513 can be provided. When selected, the full screen option 3513 may minimize or hide the entirety of the first portion 3501. This would allow expansion of the second portion 3507 so that the real-time image of the environment can be expanded and shown over a larger area. Another option (not shown) can also be provided so that the user is capable of bringing back the first portion as needed; for example, if reference to the activity instructions 3505 is needed to be performed at a later time during the activity.

[0270] As shown in FIG. 62B, after the “start” button 3506 has been selected, a notification 3514 can be provided on the display informing a user to start the activity. In various embodiments, the notification may include instructions to insert the instrument(s) to be used with the exercise into the surgical trainer. As noted above, the system needs to be able to detect theDocket No.: 10133-USP-PCTinstruments being used else the exercise cannot be started and / or be determined to be invalid / failed attempt. In various embodiments, the notification 3514 may include information about what instruments should be used.

[0271] In accordance with various embodiments, prior to the “start” button 3506 being selectable, the system determines the surgical exercise within the camera view matches the expected or predetermine surgical exercise for the selected surgical activity. In various embodiments, for example, prior to the system enabling the “start” button 3506 to be selectable, the system determines a type of gauze positioned within the camera view. If the system determines that the gauze type does not match or correspond to the expected or predetermined type of gauze, then the system generates an error, providing a notification, e.g., notification 3523, indicating that a different gauze type should be used for the particular activity. In various embodiments, for example, the notification may include instructions to ensure that a gauze with a single circle, e.g., a testing gauze, is being used. Once the system determines that the gauze type matches the expected or predetermined gauze type, then the system enables the “start” button, allowing it to be selectable. As shown, a progress portion 3524 can be provided by the system providing information or notifications that setup checks are in progress or being performed by the system, including for example, checking the gauze type.

[0272] Along with the notification 3514 that would be used to direct the user to start the activity, there is also a “finish” button 3522. The “finish” button 3522 is selectable after completion of all the steps in the activity. In various embodiments, the “finish” button 3522 may be greyed out until the computing system detects all the objects nearing an end state (e.g., all the pegs on an opposite side from a starting point). In various embodiments, the “finish” button 3522 may be greyed out for a pre-determined period of time.

[0273] When selected, the “finish” button 3522 informs the system to stop recording of the performance of the activity. In various embodiments, a prompt may appear that warns the participant that all instruments should be removed from the environment and to ensure that the activity has been fully completed after which another selection of the “finish” button 3522 would terminate the activity. In various embodiments, termination of the activity may automatically be performed a pre-determined period of time after selection of the “finish” button 3522.

[0274] In accordance with various embodiments, exemplary displays are shown in FIGS. 62D-62I configured to capture photos of the gauze types for analysis, e.g., accuracy metricsDocket No.: 10133-USP-PCTdetermination, by the system. In various embodiments, such displays or screens are provided after the precision cutting surgical activity is finished or terminated as determined by the system. In various embodiments, a real-time image capture of the environment (e.g., the surgical exercise within the surgical trainer) is displayed as provided by the system. A notification, as generated and displayed by the system, provides instructions to place the appropriate, expected, and / or predetermined gauze type within a capture frame, e.g., a non-greyed out portion and / or delimited by a greyed out portion, within the real-time image as provided by the system. For example, as shown, in FIG. 62D, a notification 3526a, provides that a circle cut-out or inner gauze portion should be placed within a circular capture frame 3527a to take or capture a photo of the circle cut-out for capture, uploading, detection, determination, and / or analysis by the system. The notification 3526a can provide additional information such as notifying the gauze should only thing be within the circular capture frame and entirely visible or not covered. Similarly, for example, as shown, in FIG. 62G, a notification 3526b, provides that a gauze square cut-out or outer gauze portion should be placed within a square or rectangular capture frame 3527b to take or capture a photo of the gauze square cut-out for capture, uploading, detection, determination, and / or analysis by the system. The notification 3526b can provide additional information such as notifying the gauze should only thing be within the circular capture frame and entirely visible or not covered. A description of instructions 3505 relevant or associated with the other displayed information and / or operation of the surgical activity can also be provided, e.g., providing guidance on how the photo of the expected gauze should be performed.

[0275] In various embodiments, a capture button 3525 is provided by the system. When selected, the system captures the image of the gauze. The capture button 3525 is greyed out or not selectable until the system determines that the appropriate, expected, and / or predetermined gauze type within the capture frame. In various embodiments, the system provides the captured image, e.g., a photo 3528a of the circle cut-out, and / or a next button 3529a is provided by the system to initiate the next step or operation of the surgical activity. In various embodiments, the system provides the captured image, e.g., a photo 3528b of the gauze square cut-out, and / or a submit button 3529b is provided by the system to initiate the next or finishing step or operation of the surgical activity. In various embodiments, with the submit button selected and detected by the system, the system uploads and / or analyzes the captured photos, e.g., photos of the circleDocket No.: 10133-USP-PCTand / or square cut-outs. In various embodiments, the system analyzes the photos to determine an accuracy metrics and / or score.

[0276] With reference to FIG. 62J, an example feedback screen is shown. The feedback screen is provided after the ‘finish’ selection is selected (see 3315; FIG. 33) during a precision cutting activity. In the example feedback screen 3700, an upload status bar 3701 is shown which provides a status update on how far along the process of uploading the recording of the video data has been completed. The upload status bar 3701 also provides a warning that informs that the video is being uploaded and that the current page should not be closed. In various embodiments, a numerical value (e.g., percentage) is provided to inform how far along the upload is at. Furthermore, the upload bar can fill up (from left to right) to correspond with the numerical value shown. In various embodiments, the numerical value may be a countdown timer which estimates how long until the upload has been completed.

[0277] In various embodiments, while the upload is being performed, information about the performance of the activity is provided. In various embodiments, the feedback is user specific. In the feedback screen 3700, a portion shows the criteria for passing the activity being performed. The criteria are all listed within a criteria box 3702 which includes threshold values for one or more individual criteria that would be considered passing. The criteria are specific to the activity; different criteria may be used for different activities. For example, as shown in the figure, example criteria include time, smoothness, and accuracy. In various embodiments, each of the criteria listed within the criteria box 3702 would need to be at or better the passing level for the activity to have been deemed passing.

[0278] The feedback screen 3700 also includes a current score box 3703. The current score box 3703 includes the current performance and corresponding valuations / scores of the different criteria. The respective scores shown in the current score box 3703 are provided by the system after an evaluation of the user’s performance. In the example, all the criteria have met or performed at a better level than the passing score threshold listed in the criteria box 3702.

[0279] The scores highlighted in the current score box 3703 are calculated based on information determined by the surgical training evaluator regarding each of the criteria specific for the surgical activity. In various embodiments, the surgical training evaluator determines metrics for the surgical activity. In various embodiments, the determined metrics comprise bimanual dexterity, smoothness, motion efficiency or tool tip path length, accuracy, and / orDocket No.: 10133-USP-PCTcenteredness. In various embodiments, the surgical training evaluator assigns the determined metric (e.g., bimanual dexterity, smoothness, tool tip path length, centeredness, or accuracy) to a score (e.g., bimanual dexterity score, smoothness score, motion efficiency score, centeredness score, or accuracy score). In various embodiments, the surgical training evaluator assigns the determined metric (e.g., bimanual dexterity, smoothness, tool tip path length, centeredness, or accuracy) to a score (e.g., bimanual dexterity score, smoothness score, motion efficiency score, centeredness score, accuracy score) using a percentage or predetermined scale, e.g., 0 to 100. In various embodiments, a high score value, e.g., near 100, is associated with better proficiency with that criterion for the activity. In various embodiments, a passing score and / or passing threshold is predetermined, e.g., 80. In various embodiments, to determine a score, for example, shown in the current score box 3703, a predetermined relationship or association between a metric value and the score are used. In various embodiments, the predetermined relationship comprises an equation, where the score = m * (metric) + b; where m and b are dependent on criteria and conditions for passing the criteria.

[0280] As an example, in accordance with various embodiments, one of the criteria determined by the system for the pegboard performance is bimanual dexterity (BD). In various embodiments, a passing score or threshold (e.g., 80) is assigned to a passing BD metric where there is a 10% difference in distance between the left and right tool and a best BD metric value corresponds to 0% difference and is assigned a score of 100. Based on a line formed between the passing score (i.e. .1,80) and best score (i.e. 0, 100), a corresponding slope = -200 and intercept = 100 is obtained. Thus, in various embodiments, the scores for BD can be determined by the surgical training evaluator using the equation (score = -200 * (metric) + 100. Example calculations are provided below:a. If a BD metric is .2 (or 20%), then the corresponding score is-200 *.2 + 100 = 60 b. If a BD metric is .1 (or 10%), then the corresponding score is-200 *.l + 100 = 80;which corresponds to a “passing” score or thresholdc. If a BD metric is .05 (or 5%), then the corresponding score is-200 *.05 + 100 = 90 d. If a BD metric is 0 (or 0%), then the corresponding score is-200 * 0 + 100 = 100

[0281] Other scores ((e.g., smoothness score, motion efficiency score, centeredness score, or accuracy score), for other respective metrics (e.g., smoothness, tool tip path length, centeredness, or accuracy), in various embodiments, can be similarly determined and displayed within theDocket No.: 10133-USP-PCTfeedback screen. Furthermore, in various embodiments, the scores are used to calculate a high score shown, for example, in the high score box 3704.

[0282] The feedback screen 3700, in the illustrated embodiment, also includes a high score box 3704. The high score box 3704 shows the one attempt with the highest “total” score. To facilitate the “total” score comparison, criteria (such as time) may need to be assigned a value on the 0-100 score value. The best score would be obtained by comparing the scores from all past attempts and selecting the one performance with the highest value. In various embodiments, the high score box 3704 is user specific. In various embodiments the high score box 3704 can be universal (i.e., covers multiple users).

[0283] The feedback screen 3700, in various embodiments, includes a graph 3705 which illustrates performance of one or more criteria over a number of different attempts. For example, the figure illustrates the graph 3705 showing the difference in time needed to complete the activity by the user over three attempts. Other criteria can also be shown graphically. In various embodiments, the system is capable of displaying each of the criteria based on user selection from the feedback screen 3700. In various embodiments, each of the criteria may be displayed in graphs 3705 altogether (e.g., one on top of each other). The graph 3705 may include a notification 3708 informing the user of how well the user performed at the current attempt compared to the previous attempt.

[0284] Once the upload bar 3701 has completely filled, the upload bar 3701 may be replaced with an upload completion notice 3710 informing that the upload has been completed within the feedback window 3700. As seen in FIG. 62K, the feedback window also includes a retry button 3711 and an exit button 3712 near the upload completion notice 3710. The retry button 3711 resets the current activity to the beginning. Setup for the activity would need to be performed (and the system confirms the setup is correct) before the activity can be repeated. The exit button 3712 terminates the current activity and allows selection of a different activity.

[0285] FIG. 62L shows an example feedback screen where a non-passing score has been determined. As seen in the figure, the feedback screen includes the upload status bar 3710, the criteria box 3702, the current score box 3703, and the high score box 3704. However, this time, the scores in the current score box 3703 do not all meet or exceed the predetermined threshold necessary for passing as highlighted in the criteria box 3702. In various embodiments, the system is configured to make these determinations and / or comparisons and provide theDocket No.: 10133-USP-PCTcorresponding display. As shown, for example, the smoothness score in the current score box is below the necessary passing score identified in the criteria box 3702. As such, the current attempt is determined to be a failure and a “try again” notice 3902 is provided to inform the user that the current attempt is non-passing.

[0286] FIG. 62M shows a similar feedback screen 4100 discussed above with sections such as the notice 4101 used to inform the status of the upload of the video, the options to retry and exit the current activity via the retry 4102 and exit 4103 buttons. The criteria box 4104, current score box 4105, and high score box 4106 are also provided. In various embodiments, some information may be missing within the criteria box 4104 and the high score box 4106 due to system errors or connection issues. With reference to 62N, retrieval of the activity and userspecific information may not be impacted, for example, if stored locally on the system.However, the notice 4108 in the feedback screen shows an issue with the uploading process (e.g., “video upload has failed”).

[0287] With reference to FIG. 620, the feedback screen shows an error 4111 associated with the photos being displayed. In various embodiments, the photos are stored in a separate location from the system. Thus, when a photo is requested, photo data is retrieved and then processed by the system to provide the photo in the feedback screen. The notice provided in the error 4111 highlights the possible cause (e.g., the inability to communicate with the storage location over the internet). FIG. 62P illustrates an embodiment whereby a current attempt was not completed. In particular, the feedback screen 4200 provides a notification 4210 that the activity time limit exceeded, e.g., 6 minutes. As noted above, each activity may have a pre-determined maximum time allotted to complete the activity. Since the activity was not completed, the current score box 4220 is left empty. In various embodiments, the current score box 4220 could provide a score up to the point in progress to the activity with a marking that the attempt was not complete. The system, in various embodiments, indicates that the feedback screen 4200 should not be closed until the upload is complete via the notification 4230. Once the upload is complete, the notification 4230 changes to inform the user and then allows the user a choice between retrying the same activity or terminating the activity (and potentially allowing selection of a different activity) 4240 as seen in FIG. 62Q.

[0288] In various embodiments, a timer notification 4810 can be provided as seen in FIG. 63. As mentioned above, the different activities that can be performed using the surgicalDocket No.: 10133-USP-PCTperformance automated assessment system have a pre-determined maximum time limit that is allotted before the system deems the attempt invalid / failed. As seen in the display screen, the timer notice 4810 provides information to the user that the maximum activity time is almost reached; which for the pegboard performance is identified as being 30 minutes 4820. In various embodiments, the maximum activity time can be adjusted to be more or less based on the exercise and / or the difficulty. In various embodiments, the timer notice 4810 may only appear at pre-determined thresholds (e.g., 15 min, 10 min, 5 min, 2 min, 1 min, etc....) and disappear until another threshold is crossed. In various embodiments, the timer notice 4810 may appear at a first pre-determined threshold and remain on the screen acting as a countdown timer indicating to the user how much time is remaining for the activity. Once the maximum activity time allotted for the activity has been reached as determined by the system, the current attempt on the activity will be determined to be incomplete.

[0289] FIG. 64 illustrates an exemplary assessment history page. The assessment history page provides activity specific information for a specific activity. As shown, the assessment history page 4900’ is specific to the precision cutting activity. From the assessment history page 4900, the surgical performance automated assessment system provides information about a user’s performance related to an activity 4909’ and the capability to start the activity 4910. The precision cutting activity 4909 information includes scores related to the highest or best performance 4911 by the user. In various embodiments, there may also be additional notes 4912 that are directed towards improving performance. In various embodiments, the additional notes 4912 may be specific to the user and / or based on past attempt scores. In various embodiments, the additional notes 4912 may be generic suggestions on how to improve one or more criteria scores. With the highest or best performance 4911 by the user, the surgical performance automated assessment system also provides the associated photos 4918 and video recording 4913, allowing, for example, a past performance tied to their best performance on the activity to be viewed. In various embodiments, any past video recording associated with any past attempt can be retrieved and displayed within the assessment history page 4900.

[0290] In various embodiments, a graph 4914 is also provided that shows user attempts over a number of different attempts. The graph 4914 shows one graph for a criteria at a time; with different graphs being displayable based on selecting one of the criteria options 4915 within the graph 4914. As shown, the current graph provides the smoothness score based on the selectionDocket No.: 10133-USP-PCTin the criteria options 4915. Furthermore, the graph 4914 shows a progression towards the passing score 4916, which in various embodiments is denoted by a dashed line. In various embodiments, the graph 4914 displays a pre-determined number of previous attempts (e.g., last 10 attempts as illustrated in the figure). In various embodiments, any subsequent attempt would remove the oldest attempt from the graph 4914.

[0291] With reference to the assessment history page 4900, information about whose information is tied to the assessment history page 4900 is shown 4908. Thus, the information shown is user-specific and, in various embodiments, would require a login to access. The assessment history page 4900 also provides the ability to go to other menus associated with other features or functions of the surgical performance automated assessment system. “Home” 4901 brings back up the main page. “Courses” 4902 brings up the different activities and / or difficulties that can be selected. “My videos” 4903 links or provides past recorded and uploaded video data of past attempts on all the activities associated with the surgical performance automated assessment system. In various embodiments, “Video Library” 4904 provides access to all past recorded and uploaded video data of the other users. “Assessment History” (as discussed above) 4905 provides information about past performances and comparison between the past performances. In various embodiments, “Resources” 4907 may include informational videos that can be viewed to teach how to perform an activity and / or skills associated with the activity.

[0292] As mentioned above, various embodiments of the feedback screen can include multiple graphics illustrating user performance for criteria over multiple attempts. For example, in FIGS. 65A-C, a feedback screen provides graphs corresponding to different criterion. The time graph 3806 highlights total time scores, e.g., how fast the activity has been completed, the smooth graph 3808 highlights smoothness scores, and the accuracy graph 3811 highlights accuracy scores.

[0293] In various embodiments, the scores are determined by the surgical training evaluator. The scores, in various embodiments, are based on metrics determined by the surgical training evaluator. For example, a total exercise time score, bimanual dexterity score, smoothness score, motion efficiency score, accuracy score, and / or centeredness score are based on or determined from corresponding respective metrics, e.g., total exercise time, bimanual dexterity, smoothness score, tool path tip length, accuracy, and / or centeredness.Docket No.: 10133-USP-PCT

[0294] In various embodiments, a passing metric value and a best metric value are predetermined. The predetermined passing metric value is assigned a passing score, e.g., 80, and the predetermined best metric value is assigned a best score, e.g., 100. Using the predetermined passing metric value, best metric value, passing score and best score, two points can be assigned (point A (passing metric value, passing score) and point B (best metric value, best score). A line can be drawn using these two points and a slope (“m”) and intercept (“b”) determined from the line. The determined slope and intercept are used in the score equation (score = m * (metric value) + b)). Using the score equation, the system upon determining a metric value for a given attempt provides the associated score.

[0295] In other words, a passing bimanual dexterity (BD) metric value and a best BD metric value are predetermined. The predetermined passing BD metric value is assigned a passing score, e g., 80, and the predetermined best BD metric value is assigned a best score, e.g., 100. Using the predetermined passing BD metric value, best BD metric value, passing score and best score, two points can be assigned (point A (passing BD metric value, passing score) and point B (best BD metric value, best score). A line can be drawn using these two points and a slope (“m”) and intercept (“b”) determined from the line. The determined slope and intercept are used in the BD score equation (BD score = m * (BD metric value) + b)). Using the BD score equation, the system upon determining a BD metric value for a given attempt provides the associated BD score.

[0296] In various embodiments, a total or overall score is determined based on other determined scores including time by the surgical training evaluator. In various embodiments, the surgical training evaluator utilizes a predetermined relationship or association, e.g., an equation or lookup table, utilizing other determined scores including time to determine the total or overall score. In various embodiments, in a total or overall score determination, one or more scores, may be weighted differently than others, i.e., one or more scores having greater influence on the overall or total score based on their assigned weights, e.g., a bimanual dexterity score being weighted higher than total exercise time.

[0297] As shown in FIGS. 66-68, in various embodiments, a simulated tissue dissection exercise for a simulated surgical activity comprises a simulated dissectible tissue pad 101. In various embodiments, the simulated dissectible tissue pad is configured to be cut by a surgical scissor, such as a laparoscopic scissor and manipulated by surgical dissectors, such as a laparoscopic dissector, and manipulated by surgical graspers, such as laparoscopic graspers. InDocket No.: 10133-USP-PCTvarious embodiments, the simulated dissectible tissue pad comprises multiple layers and in various embodiments, the simulated dissectible tissue pad comprises a first, upper layer configured to be cut into, a second, inner layer configured to be manipulated, and / or one or more anatomical landmarks and / or tubular structures, e.g., simulated vessels, identifiable through the manipulation of the second layer. In various embodiments, the simulated dissectible tissue pad comprises one or more bifurcations, e.g., branches of simulated vasculature, vessels and / or tubular structures 105. In various embodiments, one or more bifurcations are embedded in the simulated dissectible tissue pad. In various embodiments, the simulated dissectible tissue pad comprises a simulated tissue layer covering one or more bifurcations embedded in webbing, fibrous material and / or other simulated tissue layers. In various embodiments, the simulated dissectible tissue pad comprises an outer periphery 102 outlining or delimiting an inner or central portion 103. In various embodiments, one or more of the plurality of posts are attached and / or extend through the outer periphery of portion 102 and one or more bifurcations are attached or disposed within the inner portion 103. In various embodiments, the one or more bifurcations are separated from the outer portion 102 and thus, the one or more posts do not obstruct or interfere with the one or more bifurcations disposed or attached to the inner portion 103. In various embodiments, an outer simulated tissue layer and an inner simulated tissue layer of the dissectible media are configured to delimit a cavity or enclosure. In various embodiments, one or more bifurcations are disposed within the enclosure. In various embodiments, a middle or intermediate simulated tissue layer is disposed between the outer and inner simulated tissue layer and one or more bifurcations are disposed on and / or embedded in the middle simulated tissue layer. In various embodiments, an outer periphery of the outer simulated tissue layer is coupled to an outer periphery of the inner simulated tissue layer delimits and / or forms a cavity or enclosure in which one or more bifurcations disposed in the cavity or enclosure.

[0298] In various embodiments, the simulated dissectible tissue pad is removably connected to a platform and in various embodiments, the simulated dissectible tissue pad is angled relative to the platform. The platform, in various embodiments, comprises a pad support base 95 comprising a plurality of posts extending from the pad support base 95. In various embodiments, the simulated dissectible tissue pad is removably connected to one or more of the plurality of posts. In various embodiments, the simulated dissectible tissue pad has one or more apertures or openings with one or more of the plurality of posts extending through a corresponding aperture of one orDocket No.: 10133-USP-PCTmore apertures. In various embodiments, the plurality of posts have different heights arranged to support or suspend different portions of the simulated dissectible tissue pad at the height of a respective post attached to the portion of the simulated dissectible tissue pad. In various embodiments, four posts arranged near an outer periphery of the pad support base extend vertically or perpendicularly from the pad support base in which two posts, e.g., two distal posts 94a, are taller or longer than the two other posts, e.g., two proximal posts 94b. In various embodiments, the simulated dissectible tissue pad is configured to have an outer periphery smaller than an inner and / or outer periphery of the platform, e.g., pad support base 95, facilitating the stability and / or operation of the simulated dissectible tissue pad.

[0299] In various embodiments, the tissue dissection surgical exercise comprises a simulated dissectible tissue pad 101 with one or more bifurcations of one or more simulated vasculature hidden or not visible. In various embodiments, the tissue dissection surgical activity begins with simulated vasculature including all bifurcations (e.g., white and red bifurcations) hidden and continues with dissecting, grasping, and retracting tissue to clear an area and expose each bifurcation so it becomes visually distinguishable from the background, e.g., other portions of the simulated dissectible tissue pad.

[0300] In various embodiments, a tissue dissection surgical activity utilizing a simulated dissectible tissue pad 101 is provided to allow learners to repeatably and objectivity practice dissecting, grasping, and retracting surgical techniques. In various embodiments, a tissue dissection surgical activity is provided to be assessable as to proficiency of dissecting, grasping, and / or retracting surgical techniques utilized. In various embodiments, the system upon determination of completion of the surgical activity provides access to others to assess the surgical performance, e.g., proficiency of surgical skills utilized, and / or to provide formative feedback by providing a video of the surgical activity. In accordance with various embodiments, the surgical training evaluator utilizing semantic segmentation, contour determination, and / or instrument tracking is configured to determine one or more metrics of a user’s performance of a tissue dissection surgical activity. In various embodiments, such metrics include temporal metrics, e.g., total exercise time, and / or motion metrics, e.g., tool tip path length, motion smoothness, and / or bimanual dexterity, derived from timing and / or surgical instrument tracking throughout the surgical activity, referring respectively to time-based and / or motion-based aspects of a surgical activity. In various embodiments, these metrics provide insights into the efficiency and pacing ofDocket No.: 10133-USP-PCTa user’s, e g., a surgeon’s, actions and / or a detailed analysis of the surgeon's movements, offering insights into their precision, efficiency, and dexterity. In various embodiments, recorded videos are used to allow non-real time analysis and to minimize extensive computational power. In various embodiments, the system assists others, e.g., a faculty member, to determine, assess, and / or evaluate a learner’s surgical proficiency. In various embodiments, the system is configured to determine, detect and / or identify when the surroundings of bifurcations are cleared and distinguishable from the background. In various embodiments, the system is configured to detect, mark, record and / or identify specific moments in the video in which when surroundings of bifurcations are cleared and distinguishable from the background, and, in various embodiments, this enables others, e.g., faculty, to quickly access and review the necessary or specific portions of the video to streamline the assessment or grading process, significantly reducing the time required for faculty to evaluate a learners' performance. In various embodiments, the system identifies and / or highlights such specific moments in the video when a bifurcation becomes visible and / or cleared to an assessor, e.g., not the learner, for example, via annotations, e.g., via an annotation bar 7113, in the video or media controls and / or other via other informational form data, e.g., in a bifurcation analysis portion or section 7104 as shown in exemplary displays in FIGS. 71A-C. In various embodiments, the system does not necessarily assess the skill of learners, but rather facilitates others, e.g., faculty, who will assess the learners' performance.

[0301] With reference to FIG. 69A, an exemplary display is shown after the setup of the simulated surgical exercise, e.g., a simulated tissue dissection exercise, has been checked / verified by the system ((operation 3305) as illustrated in FIG. 33). In various embodiments, the exemplary display corresponds to after the setup of the surgical exercise within the surgical trainer and determination that the setup is correct via the notification “setup check complete” 3508.

[0302] As shown, the display comprises two portions or sections. A first section (on the right) 3501 contains information about the surgical activity (e.g., Tissue Dissection). A second portion (on the left) 3507 displays the real-time image capture of the environment (e.g., the surgical exercise within the surgical trainer). In various embodiments, the first portion 3501 provides information about the surgical activity as supplied by the system. The first portion 3501 provides information such as the name 3502 of the activity (e.g., Tissue Dissection), a maximum video / exercise record time 3503’ for the activity (e.g., 30 minutes), and / or criteria indicating a passing score and / or condition. In various embodiments, the first portion 3501 provides a list of activityDocket No.: 10133-USP-PCTinstructions providing details what cannot be done during the activity lest an invalid attempt be determined and / or how to perform the activity properly from the beginning to the appropriate end state. In various embodiments, such instructions may include information that a hand needs to be kept out of the environment, that the instruments should be maintained within the environment, that the performance of the activity or recording should not exceed a pre-determined period of time, and / or that the activity has not been properly finished. In various embodiments, the list of activity instructions can also include instructions on how to perform the activity properly from the beginning to the appropriate end state. In various embodiments, a start button 3506 is provided. When selected, the system starts or initiates the selected activity. The “start” 3506 button may be greyed out or not selectable until confirmation that the system setup checks are verified, complete, or confirmed. In accordance with various embodiments, prior to the “start” button 3506 being selectable, the system determines the surgical exercise within the camera view matches the expected or predetermine surgical exercise for the selected surgical activity.

[0303] In various embodiments, prior to the “start” button being provided, a “setup check” button is provided. In various embodiments, the “setup check” button is greyed out, disabled, or not selectable until confirmation by the system that predetermined checks or steps are performed, verified, and / or confirmed. In various embodiments, the first portion 3501 provides setup information and / or instructions for the surgical activity as supplied by the system. In various embodiments, the system detects if or when a surgical trainer and / or a camera is connected and / or displays a video feed and / or image from the connected camera to the second portion 3507. In various embodiments, once the system detects if or when a surgical trainer and / or a camera is connected and / or displays a video feed and / or image from the connected camera to the second portion 3507, the “setup check” button is enabled or selectable. In various embodiments, once the “setup check” button is selected or initiated, the system starts or initiates the system determines the surgical exercise within the camera view matches the expected or predetermined surgical exercise, placement and / or arrangement for the selected surgical activity. In various embodiments, after the “setup check” button has been selected, a notification can be provided on the display informing a user that the system is checking the setup of the surgical activity and / or exercise. In various embodiments, the notification may include instructions to remove instrument from the camera view or environment, tilt or fully tilt the surgical trainer, and / or insert, arrange and / orDocket No.: 10133-USP-PCTposition the expected or predetermined surgical exercise for the selected, associated and / or expected surgical activity within the camera view.

[0304] In various embodiments, once the “setup check” button is selected, a video and / or image from the camera is recorded and / or a record created and / or associated within a storage database. In various embodiments, the record comprises parameters and / or attributes, including, for example, a user, user group, and / or activity. In various embodiments, once a “start button” is provided and / or selected, the recording of the video and / or image from the camera associated or initiated with the setup check is stopped, saved, and / or uploaded to memory storage.

[0305] In the illustrated embodiment, the second portion (on the left) 3507 of the display, the real-time image of the environment associated with the activity to be performed (e.g., tissue dissection) is shown. The real-time image is acquired by the camera of the surgical trainer. In various embodiments, the real-time image displays at least a portion of the surgical exercise within a framed view that allows adequate viewing of the surgical exercise on the display. In various embodiments, a full screen option 3513 can be provided. When selected, the full screen option 3513 may minimize or hide the entirety of the first portion 3501. This would allow expansion of the second portion 3507 so that the real-time image of the environment can be expanded and shown over a larger area. Another option (not shown) can also be provided so that the user is capable of bringing back the first portion as needed; for example, if reference to the activity instructions is needed to be performed at a later time during the activity.

[0306] As shown in FIG. 69B, after the “start” button 3506 has been selected, a notification 3514 can be provided on the display informing a user to start the activity. In various embodiments, the notification may include instructions to insert the instrument(s) to be used with the exercise into the surgical trainer. In various embodiments, the system is configured to detect that instruments being used and / or within the camera view else the exercise cannot be started and / or be determined to be invalid / failed attempt. In various embodiments, the notification 3514 may include information about what instruments should be used. In various embodiments, once the system determines that the activity has been started, e.g., instruments are detected within the camera view, the system starts or initiates a recording of the video, image, and / or feed from the camera and / or a surgical activity timer.

[0307] In various embodiments, along with the notification 3514 that would be used to direct the user to start the activity, the system provides a “finish” button 3522. In various embodiments,Docket No.: 10133-USP-PCTa “finish” button is provided, enabled, and / or selectable, once the system determines that the expected surgical activity has been started, as shown for example in FIG. 69C. The “finish” button 3522, in various embodiments, is selectable after completion of all the steps in the activity as determined by the system. In various embodiments, the “finish” button 3522 may be greyed out until the system detects the activity nearing an end state. In various embodiments, the “finish” button 3522 may be greyed out for a pre-determined period of time.

[0308] In accordance with various embodiments, if the system does not detect instruments within the camera view and / or environment within a predetermined time period, e.g., 1 minute, a notification is provided to reinsert the expected instrument or the attempt is marked, recorded and / or determined to be invalid or failed. If the system fails to detect instruments within the camera view within the predetermined period of time, in various embodiments, the system marks and / or records the attempt to be invalid or failed and / or a recording of the camera feed is stopped, saved and / or uploaded.

[0309] When selected, the “finish” button 3522 informs the system to stop recording of the performance of the activity and / or an initiation of saving and / or uploading of the recorded performance for storage and / or post analysis, as shown for example in FIG. 69D. In various embodiments, a prompt or notification 3514 may appear that warns the participant that all instruments should be removed from the environment and to ensure that the activity has been fully completed and / or in various embodiments, another selection of the “finish” button 3522 would terminate the activity. In various embodiments, termination of the activity may automatically be performed a pre-determined period of time after selection of the “finish” button 3522 and / or once predetermined conditions are met, e.g., the system does not detect instruments within the camera view or environment. In various embodiments, termination of the activity may automatically be performed a pre-determined period of time after selection of the “finish” button 3522.

[0310] In various embodiments, once the system determines that the surgical activity is finished or terminated based on one or more predetermined conditions, e.g., the selection or initiation of the “finish” button and / or expiration of an activity timer or predetermined time period, the system stops recording of the performance of the activity, e.g., the video / image and / or camera feed, and / or an initiation of saving and / or uploading of the recorded performance for storage and / or post analysis. In various embodiments, a notification or prompt is provided including, for example, a progress bar with a percentage displayed providing how much of the video has beenDocket No.: 10133-USP-PCTsaved and / or uploaded, an amount of the video data saved and / or uploaded, a total video size, and / or an estimated time remaining for saving and / or uploading the video. In various embodiments, a “cancel” button is provided to pause and / or terminating the saving and / or uploading of the video. In various embodiments, a “download” button is provided by the system to save the video to a specific or predetermined location.

[0311] In various embodiments, once saving, uploading and / or video processing are performed and completed by the system, the system provides a submission page, screen and / or display, as shown, for example, in FIG. 69E. In various embodiments, the system provides the video and access or controls to play, pause, scrub through video, and / or increase / decrease playback speed. In various embodiments, length of video, current play time, activity name and / or course title are provided by the system. In various embodiments, the system provides one or more submission options, e.g., a default submit for assessment option with a submission destination or designator, e.g., a proctor (default) or peer, and / or a sharing or posting of the video to a public or open video library. In various embodiments, the system provides a “save to my videos” option and if selected, the submission destination or designator options are removed. In various embodiments, the system provides a retake option and if selected, the system ignores any submission options, saves the video to a private, user, and / or predetermined video library and / or no video is submitted for assessment. In various embodiments, the system initiates or enables the setup check stage or process for the selected activity, if the retake option is selected or initiated.

[0312] In various embodiments, the system provides a submit option and if selected, the system marks, records, and / or identifies the video to be submitted for assessments based on the submission options selected or otherwise predetermined. The system, in various embodiments, identifies any errors is submission option selections and / or indicators to correct the identified errors. In various embodiments, once the system initiates the submit option, if a sharing or posting of the video to a public or open video library and / or a “save to my videos” option are selected, the system performs the designated selected options.

[0313] In various embodiments, the system provides a different actor, process or system to assess a surgical activity video. In various embodiments, the system provides an assessment of a tissue dissection activity, e.g., a video submitted by a resident for assessment by a faculty, program director, and / or proctor. In various embodiments, one or more submitted videos for assessment from one or more sources are provided by the system. In various embodiments, the systemDocket No.: 10133-USP-PCTprovides access and selection of one or more submitted videos as shown for example in FIG. 70, illustrating an exemplary assessment video queue page 7001. In various embodiments, the system provides access, selection and / or assessment of one or more submitted videos to a system or entity different from the system, process or entity that submitted, saved, uploaded, and / or recorded the video. As shown, the system provides access to submitted recorded videos of surgical activities 7002 from one or more sources to be accessed and assessed by different sources, i.e., a source not / different from the source that submitted the video. Informational data 7003 of each of the submitted recorded, such as its source, title, duration, date, and / or assessment score can also be provided. The system provides a start assessment button / option 7004 and if selected, the system retrieves the submitted video and prepares and provides an assessment form page, display or screen, as shown for example in FIG. 71 A.

[0314] In various embodiments, as shown, for example, in FIG. 71 A-C, the system provides a global rating scale (GRS) form portion or section 7101 to score, assess, and / or rate the submitted video 7102. In various embodiments, the system provides real-time automatic overall skill level assessments. As shown, for example, in FIGS. 71 A and 73, an overall skill level portion or section 7107 is provided by the system in which the automatic or automated assignment or display of the overall skill level, e.g., novice, competent, and advanced, is shown.

[0315] In various embodiments, the system automatically determines an assessment and / or assigns a score, level and / or rating of a surgical skill. In various embodiments, the determination of the assessment and / or scores, level and / or rating is automatic or automated, including no or limited user interaction, and based on a global rating dataset of thousands of surgical skill assessments, thereby reducing faculty workload and / or enhancing consistency in skill evaluation. In various embodiments, the assigned overall surgical skill levels are novice, competent, and advanced, e.g., levels in overall skill level portion 7107, and / or based on global rating scale (GRS) scores, including but not limited to scores assigned to the current actively assessed surgical activity, e.g., scores in GRS form portion 7101, and a plurality of other scores assigned to previously assessed surgical activity, e.g., a global rating dataset.

[0316] Referring also to FIGS. 72 and 73, the selected or assigned GRS scores 7201, e.g., via input provided and shown in a GRS form portion or section 7101 are gathered by or provided to the overall skill level assessment system 7202 which retrieves an overall skill level model from a content delivery network (CDN) 7203 based on the particular surgical activity, e.g., tissueDocket No.: 10133-USP-PCTdissection. The overall skill level assessment system predicts the overall skill level using the selected scores 7201 and overall skill level model 7204, providing, returning or displaying the result, the predicted overall skill level 7205, in, for example, the overall skill level portion or section 7107. In various embodiments, the system is configured to allow the modification, overriding, and / or overwriting of the predicted overall skill level and provided by the system by the assessor via selecting a different overall skill level in overall skill level portion or section 7107.

[0317] In various embodiments, an overall surgical skill level is determined by the system using a global rating dataset that is generated or collected. In various embodiments, a global rating dataset comprises a plurality of surgical simulation assessments, e.g., over 5 thousand assessments, of one or more surgical activities. In various embodiments, the global rating dataset comprises numerous global rating scale (GRS) scores and overall skill evaluations from sources, e.g., experts such as program directors, attendings, fellows, and surgical faculty. In various embodiments, an overall surgical skill level, e.g., novice, competent, and / or advanced, are determined by the system using an overall skill level model, e.g., overall skill level model 7204, trained on the global rating dataset using algorithms such as Logistic Regression, KNN, SVC, Kernel SVM, Naive Bayes, Random Forest, and Decision Trees.

[0318] The overall skill level model 7204, in various embodiments, is configured to support several surgical simulation foundation courses, including Laparoscopic Instrumentation, Tissue Dissection, First Entry, and Laparoscopic Suturing, with simulated surgical activities such as Pegboard Performance, Petal Threading Performance, Extracorporeal Knot-Tying Performance, Running Suture Performance, Intracorporeal Knot-Tying Performance, Setting, Loading, and Driving the Needle Performance, Veress Needle Performance, Hasson Cut-Down Performance, Optical Trocar Entry Performance, Dissection Performance, Laparoscopic Vaginal Cuff Closure Performance, Contained Tissue Extraction Performance, Ovarian Torsion and Ovarian Cystectomy Performance, Total Laparoscopic Hysterectomy Performance, Vaginal Hysterectomy Performance, Laparoscopic Salpingo-Oophorectomy Performance and / or Anterior Colpotomy Performance including historical GRS data, e.g., scores and / or associated overall skill levels. In various embodiments, the overall skill level model is trained on historical GRS data using machine learning algorithms such as Logistic Regression, KNN, SVC, Kernel SVM, Naive Bayes, Random Forest, and Decision Trees. After developing the overall skill level models, the trained models were stored in a content delivery network (CDN) 7203 and accessed by the backend as needed. InDocket No.: 10133-USP-PCTvarious embodiments, the models are hosted in the cloud with backend inference to ensure security. In accordance with various embodiments, the overall skill level assessment system and / or model streamline the assessment process, reduces faculty workload, and improves evaluation consistency.

[0319] Referring again to FIGS. 71 A-C, as shown, the recorded video 7102 is currently being processed, as information about the attempt is retrieved from a stored memory location and configured for display on the display. In various embodiments, the surgical performance automated assessment system is configured to format the recorded video to be displayed on the screen, display and / or monitor. Beneath the recorded video is a media control feature 7103 which receives input regarding playing and pausing the recorded video. Details about the length of the recorded video are provided in the media control feature, in accordance with various embodiments. The surgical performance automated assessment system further provides details or data of what time the current video is at and allows jumping to various points between the start and end of the recorded video.

[0320] In various embodiments, as also shown, the system provides or displays a bifurcation analysis portion or section 7104 and in various embodiment, provides a timestamp, number, and / or color of a bifurcation as they appear, are revealed or identified during the performance and / or analysis of the activity or video. In various embodiments, the system provides an identified or revealed bifurcation to be selectable, and, in various embodiments, a current video timestamp is updated to the bifurcation reveal or identification timestamp. In various embodiments, the system analyzes the submitted video to detect, identify, and / or record a timestamp, number, and color of a bifurcation as they appear, are revealed or identified during the performance and / or analysis of the activity or video. In various embodiments, the system provides an annotation bar 7113, and, in various embodiments, the annotation bar is the same width as a video’s scrub bar, e.g., a media control feature 7103. In various embodiments, the system provides an annotation or identifier placed on or included with the annotation bar based on a timestamp of a bifurcation reveal. In various embodiments, the annotation or identifier provides a bifurcation color and a reveal number. In various embodiments, the system provides an annotation or identifier to be selectable, and, in various embodiments, a current video timestamp is updated to the bifurcation reveal or identification timestamp associated with the selected annotation. In various embodiments, theDocket No.: 10133-USP-PCTsystem provides recommended and / or additional videos that may assist in the assessment of the recorded video and / or the surgical activity and / or skill being assessed.

[0321] In various embodiments, the system is configured to provide and analyze a tissue dissection surgical activity, e.g., identify bifurcations. In various embodiments, the system is configured to detect and / or identify a type or characteristic of a bifurcation. In various embodiments, the system is configured to determine a number or tally of bifurcations revealed or located during a tissue dissection surgical activity. In various embodiments, the system is configured to determine and / or assess a surgical performance of revealing and / or locating bifurcations without causing damage to the bifurcations and / or simulated tissue or vasculature attached or proximate thereto. In various embodiments, the system is configured to identify the moment or instance of a predetermined event or condition, e.g., a bifurcation becomes visible during a surgical training activity, e.g., tissue dissection surgical activity, by, for example, analyzing a video of the surgical training exercise.

[0322] As shown, in FIG. 74, the system, in various accordance with various embodiments, analyzes a video feed, e.g., a recorded video, of a tissue dissection surgical activity, 741. The system determines and / or identifies when a bifurcation becomes visible during the surgical activity 743. In various embodiments, the system obscures and / or ignores a portion of the video feed where the bifurcation was identified in subsequent analysis of the video feed. For example, upon determination and / or identification of a bifurcation, the system creates a mask 744 and prior to the bifurcation identification 743, the system applies the mask 742 to the video feed being analyzed by the system. For each bifurcation identified, a mask is created and applied to the video feed and the modified video feed is analyzed by the system to identify any further bifurcations uncovered in the surgical activity. As such, the system does not re-identify a bifurcation it previously identified, accelerating processing times and minimizes computational power and / or resources. Prior to the identification of a bifurcation, e.g., the first bifurcation of the video feed, the system skips the operation of applying the mask 742, e.g., doesn’t apply a mask, obscure and / or ignore a portion of the video, until a bifurcation is identified. The process continues until the video feed ends, e.g., the system has analyzed the entire recorded video.

[0323] Referring to FIG. 75A-L, exemplary displays are provided to illustrate the identification, marking, and / or recording of bifurcations as performed by the system, in accordance with various embodiments. The exemplary displays are not displays or views of images or videoDocket No.: 10133-USP-PCTfeeds visible by the user, but exemplary displays illustrating operations or process performed by the system in bifurcation analysis for a simulated tissue dissection surgical activity in accordance with various embodiments. Upon identifying a bifurcation 111, the system creates a mask 112, e.g., a shape having an outline and interior that is identical or equivalent in dimensions to the identified bifurcation. In various embodiments, the system provides or assigns the mask a predetermined identifier or characteristic, e.g., a coloring, e.g., black. The system applies the mask on the identified bifurcation, effectively covering or occluding the identified bifurcation with the mask. In various embodiments, the system overlays the mask over the region of the video, as determined by the system, of which the bifurcation was identified, occluding the identified bifurcation from the camera or recorded video feed and preventing subsequent identification or recognition in subsequent analysis of the video feed, e.g., the frames of the camera or recorded video feed. In various embodiments, the system continues to analyze the video feed and for each new bifurcation identified or detected, the video timestamp is identified and recorded, and / or mask or region identified, created and applied to the video feed and subsequent analysis of the video feed.

[0324] In various embodiments, the system utilizes instance segmentation to provide and analyze a tissue surgical activity. The system, in various embodiments, is configured to detect complex objects, e.g., multi-sided polygons having differing lengths and widths, and in differing conditions such as dynamic lighting or partially obscured views. In various embodiments, the system is configured to utilize instance segmentation to detect complex objects, e.g., bifurcations. However, instance segmentation can require heavy computational power and thereby reducing the ability to provide real-time feedback, video processing and analysis taking extensive length of time and extended based on the size of the video being processed. In various embodiments, the instance segmentation process is provided via computing devices or resources separate from the initiating system or process to leverage greater and / or scalable computational power to reduce processing time and requirements by the initiating system.

[0325] In various embodiments, the system performs instance segmentation using an algorithm, e.g., YOLACT++, to detect bifurcations. In various embodiments, in preparation of utilizing instance segmentation, a diverse and representative dataset, e.g., a tissue dissection dataset, is created of images containing the objects of interest, e.g., bifurcations, in various poses, lighting conditions, and backgrounds within the confines or context of the surgical exercise, e.g.,Docket No.: 10133-USP-PCTtissue dissection exercise. In various embodiments, a tissue dissection dataset is created or generated utilizing a plurality of videos of numerous tissue dissection activities utilizing a tissue dissection exercise with each of the plurality of videos spilt into numerous single images. For each image, one or more objects of interests, e.g., a bifurcation of a simulated vessel or vasculature, are annotated. For example, each bifurcation of a simulated vasculature visible in an image is outlined, e.g., polygons are created or formed outlining, bounding, and / or encompassing a portion of each visible simulated vasculature, e.g., a bifurcation, closing or tightly fitting around the boundaries of each bifurcation of each visible simulated vasculature. In various embodiments, each polygon created for a given image is unique in location relative to the image in which no polygons overlap each other. Additionally, for each polygon, a class label is assigned indicating a category of the object of interest, e.g., a red bifurcation or a white bifurcation.

[0326] In various embodiments, data augmentation is applied to the annotated data, e g., the created and labeled polygons, increasing diversity of the dataset without collecting new data. In various embodiments, the images of the video are modified and then re-annotated. For example, the images are rotated or flipped to create or simulate different surgical exercise variations, scaled and / or cropped to create or simulate different object sizes and / or positions, and / or adjusted in terms of col or jittering, brightness, contrast, and / or saturation to account for different lighting conditions. Data labeling, for instance segmentation involves annotating each object in the images with a mask that outlines its boundaries. Data augmentation is used to increase the diversity of the training dataset without collecting new data.

[0327] In various embodiments, the system supplies the annotated and augmented data to an Al model, e.g., Mask R-CNN, YOLACT, and YOLO, and, in various embodiments, a YolactEdge model, is used. The Al model is trained using the annotated and augmented data and internal settings are adjusted to predict detection and / or determination of bifurcations in new images. The Al model’s performance is evaluated using a validation dataset and metrics, such as Mean Average Precision (mAP) (measuring the accuracy of object detection and segmentation), and Intersection over Union (loU) (evaluating the overlap between predicted and ground truth masks), are determined and / or calculated. Additionally, in various embodiments, the Al model is tested using new images, images not previously provided to the Al model, to check if the training and / or learning process of the Al model is operating as expected. In various embodiments, based on the evaluation results, the Al model can be fine-tuned to improve its performance, involving, forDocket No.: 10133-USP-PCTexample, adjusting hyperparameters, augmenting the dataset further, and / or refining the annotations.

[0328] In various embodiments, the system identifies and delineates an object of interest from a video feed. In various embodiments, the video feed is a recorded video of a surgical activity, e.g., tissue dissection, submitted by a user. In various embodiments, the system is configured to identify and delineate an object of interest in an image from a video feed. In various embodiments, the system is configured to distinguish between different objects of the same class utilizing instance segmentation.

[0329] As shown in FIGS. 76A-B, in various embodiments, a ligating loop exercise 40e comprises a ligation media 765. In various embodiments, the ligation media is positioned on a platform and is visibly distinguishable from the platform. The ligation media, in various embodiments, is a flat or planar sheet and / or has a comb-like shape with a top spine 760 and one or more fingers 761 extending from the top spine. In various embodiments, the ligation media comprises a top spine 760 with a plurality of elongate fingers spaced from each other and extending from the top spine. The top spine, in various embodiments, has a width greater than a width of a finger and / or collectively all the fingers and / or spaces therebetween.

[0330] In various embodiments, the ligation media 760 is removably connected to a platform. The platform comprises a ligation base 73. In various embodiments, the ligation media is configured to have an outer periphery smaller than an inner and / or outer periphery of the platform, e g., ligation base. In various embodiments, a portion of the ligation media is removable connected to the ligation base with other or the remaining portion of the ligation media remains free or unattached to the ligation base. In various embodiments, the top spine 760 of the ligation media is removably connected to the ligation base and the fingers are not connected, free and / or unattached to the ligation base. In various embodiments, the finger connected to the top spine removably connected to the ligation base is angled or tapers towards the ligation base and / or touches or contacts the ligation base. In various embodiments, one or more clips or connectors 74 are fixedly attached to the ligation base. The one or more clips or connectors are configured to releasably attach the ligation media to the ligation base. In various embodiments, an upper clip 74a is configured to releasably secure a first or top portion of the ligation media, e.g., a top spine 760, to the ligation base 73. In various embodiments, the ligation base 73 is removably attached to the exercise insert 50 that is removably attached to the base 14 of a surgical trainer.Docket No.: 10133-USP-PCT

[0331] In various embodiments, a finger of one or more fingers 761 comprises a ligation indicator 762 indicating and / or providing a predetermined location for placement, tying and / or knotting of a string 763, thread, suture or the like via, for example, a ligation loop tool 764. In various embodiments, the ligation indicator has a width equal to and / or thinner than a width of the string. In various embodiments, the string 763 is configured to cover, surround, and / or obscures the ligation indicator. In various embodiments, the ligation indicator is not visible and / or hidden with the string 763 covering the indicator and / or knotted around a finger 761 with the ligation indicator 762. In accordance with various embodiments, the string has length greater than, e.g., at least two times, the width of a finger 761. In various embodiments, the string is configured to wrap around, tied, and / or knotted to a portion of a finger. In various embodiments, a ligation indicator is a straight line or marking extending across the width of a finger, e.g., a central finger in the illustrated embodiment, and extending perpendicular relative to the length of the finger and / or parallel to a proximal or bottom edge of the ligation base 73 and / or clip 74a. In various embodiments, a plurality of ligation indicators are provided. In various embodiments, one or more indicators are provided on the same finger and / or one or more fingers have one or more indicators provided thereon.

[0332] In various embodiments, the ligation media is made of a compliant and / or resilient material, e.g., foam, and / or has a color and / or another visible distinguishing feature different from the ligation base 73. In various embodiments, the ligation media is flat and / or planar and has a uniform thickness. In various embodiments, the ligation media is configured to be manipulated by a surgical instrument, e.g., a grasper and / or ligation tool, and to withstand compression and / or elastic deformation due to a string being wrapped, tied and / or knotted around a portion of the ligation media, e.g., a finger 761. In various embodiments, the ligation media is configured to return to its initial, e.g., undeformed and / or uncompressed state, condition or state, with removal of a string wrapped, tied and / or knotted around a portion of the ligation media, e.g., a finger 761, and / or manipulation by a surgical instrument, e.g., a grasper and / or ligation tool. In various embodiments, the ligation media is configured to be manipulated by a surgical instrument, e.g., a grasper and / or ligation tool, and to have one or more strings or the like wrapped, tied, and / or knotted around a portion of the ligation media, e.g., along a portion of a finger along a ligation indicator disposed thereon.Docket No.: 10133-USP-PCT

[0333] In various embodiments, the surgical training evaluator is configured to utilize the tilt indicators and / or hand detection to verify a setup state or condition of a surgical activity utilizing a ligation loop surgical exercise. In various embodiments, a surgical grasper and / or a multi-use ligation loop tool is used to perform the ligation loop surgical activity. In various embodiments, the surgical training evaluator tracks the instruments, e.g., grasper and / or ligation loop tool, to determine or infer a state of the ligating loop surgical activity. In various embodiments, the instruments, e.g., grasper and / or ligation loop tool, are not tracked, increasing assessment and / or reducing computational power and / or utilization. In various embodiments, the time to complete the ligation loop surgical activity is determined and / or recorded.

[0334] In various embodiments, a string is knotted around a finger of the ligation loop surgical exercise and is a required operation to complete the ligation loop surgical activity. Additionally, in various embodiments, once the knot is completed, the ligation loop tool is to be pulled to the left or right, removing the slack from the string and / or prevents left-over or excessive portions of the string from covering portions of the ligation media and / or base.

[0335] In various embodiments, a photo of the resulting or completed knot is provided to the system. In various embodiments, a capture button is provided by the system. When selected, the system captures the image of the ligation media or portions thereof. In various embodiments, the system provides the captured image, e.g., a photo of the ligation media or portions thereof, and / or a next button is provided by the system to initiate the next step or operation of the surgical activity. In various embodiments, with the submit button selected and detected by the system, the system uploads and / or analyzes the captured image. In various embodiments, the system analyzes the image of the ligation media to determine one or more metrics, rating, and / or score.

[0336] In various embodiments, the system determines how quickly the knot tied to the ligation media was completed, how tight the knot is, i.e., the tightness of the knot, and / or the accuracy of the placement of the knot. In various embodiments, the determination of how quickly the knot was completed is based on the total time to complete the ligating loop surgical activity, e.g., when the system detects all instruments have been removed and / or no longer detected within the camera view.

[0337] In various embodiments, to determine and / or measure the tightness of the knot, the surgical training evaluator generates a binary mask and a distance transform algorithm. In various embodiments, the system creates a binary mask of a portion the ligation media 771, e g., a portionDocket No.: 10133-USP-PCTof the ligation media, e.g., a finger, including a string knotted thereto, as shown for example in FIG. 77A, providing a mask of the center column, i.e., finger of the ligation media with the knot. In various embodiments, the surgical training evaluator using semantic segmentation generates a binary mask to identify and / or isolate the ligation media and / or portions thereof from the rest of the camera image. The generated binary mask, e.g., a digitized version, is used by the surgical training evaluator to determine or infer a state of a current activity and / or to recognize and / or track an object. In accordance with various embodiments, target objects, objects predefined to be objects of interest, are configured to have distinct colors, e.g., red, from each other, allowing for accurate and efficient segmentation of the objects by the surgical training evaluator. Although object identification or semantic segmentation as described, for example, in terms of color, other representations, e.g., two dimensional numerical values or symbols, defining unique and individual AOIs could be utilized.

[0338] In various embodiments, the surgical training evaluator further isolates a portion of the generated binary mask to determine or infer a state of ligating loop surgical activity and / or to recognize and / or track a configuration or state of a portion of the ligation media. In various embodiments, the surgical training evaluator isolates the center column or finger from the rest of the binary mask by determining the vertical and / or horizontal start and end of the columns or fingers. In various embodiments, for example, for each row of the binary mask, the number of times the row turns from off to on (or zero to non-zero) is determined. In various embodiments, with three columns or fingers, the vertical range is determined by finding the first and last row that has three transitions from off to on. The horizontal start and end of each column, in various embodiments, is determined by using the midpoint of the first and second transitions, and the midpoint of the second and third transitions in that row. Using the determined vertical and horizontal ranges, the surgical training evaluator forms a box, area, or region around the center column and the rest of the image outside the box can be ignored by the system, thereby isolating the center column or finger from the rest of the ligation media.

[0339] Utilizing the binary mask, for every point, the distance from that point to the edge of the mask 772 is measured, e.g., a distance from each pixel to the nearest non-zero pixel. FIG. 77B illustrates an exemplary distance transform of the mask in which brighter pixels represent a farther distance to the edge of the mask and in which the distance from a point or pixel to a pixel at the edge of the mask is measured. Utilizing the distance transform, the system, in variousDocket No.: 10133-USP-PCTembodiments, determines and / or measures how the width of the column changes from top to bottom. As shown in FIG. 77C, the line 773 highlights the points or pixels that are farthest from either edge as determined or calculated by finding the pixel with the largest distance from the mask edge for each row in the mask. In various embodiments, the system checks each point or pixel along the line 773 to determine which has the shortest distance to the edge of the mask. The resulting or identified point represents the narrowest point along the length of the column, e.g., finger, indicating or determining where the knot tied to the finger is located.

[0340] If the width at that point as determined by the distance from the edge of the mask is below a fixed or predetermined threshold, then the system determines that the knot is tightened correctly, e.g., the tightness of the knot is identified and / or recorded as passed or acceptable by the system for the surgical activity as having a required, acceptable, or passable tightness. In various embodiments, the width threshold is less than 50 pixels. In various embodiments, the width threshold is adjustable to vary training difficulties and / or based on aggregated, collected and / or predicted data sourced from other similar training procedures and / or users.

[0341] In various embodiments, to determine and / or measure the location, placement, and / or accuracy of the placement or location of the knot, the system determines the vertical height of the knot point 775. In various embodiments, the vertical height is the row within the determined vertical range starting from the first and last row that has three transitions from off to on to the row in which the knot point 775 is located.

[0342] In various embodiments, the system compares the determined vertical height of the knot point 775 to one or more fixed or predetermined thresholds, e.g., vertical height thresholds as represented by the green lines 774, 776, as exemplarily shown in FIG. 77D. If the system determines that the determined vertical height of the knot point is between the predetermined threshold, the system determines that the knot is correctly or accurately placed or located, e.g., the accuracy of the knot placement is identified and / or recorded as passed or acceptable by the system for the surgical activity as having a required, acceptable, or passable accuracy. In various embodiments, the vertical height thresholds 774, 776 are 300 and 500 pixels, respectively, e.g., from the top of FIG. 77D. In various embodiments the vertical height thresholds are adjustable to vary training difficulties and / or based on aggregated, collected and / or predicted data sourced from other similar training procedures and / or users.Docket No.: 10133-USP-PCT

[0343] In various embodiments, the tightness of the knot and / or accuracy of the placement of the knot for the surgical activity is a binary determination, e.g., pass or fail. Either the knot is tight, or it is not. Either the knot is placed correctly, or it is not. In various embodiments, a range, percentage and / or a numerical value or measurement unit are provided to indicate or provide a metric of the tightness of the knot and / or accuracy of the placement of the knot as determined by system.

[0344] With reference to FIG. 78E, an exemplary display is shown after the setup of the surgical exercise has been checked by the system ((operation 3305) as illustrated in FIG. 33). In various embodiments, the exemplary display corresponds to after the setup of the surgical exercise within the surgical trainer and determination that the setup is correct via the notification “setup check complete” 3508.

[0345] As shown, the display is split into two portions. A first portion (on the right) 3501 contains information about the exercise (e.g., Ligating Loop). A second portion (on the left) 3507 displays the real-time image capture of the environment (e.g., the surgical exercise within the surgical trainer).

[0346] With the first portion (on the right) 3501, the information included on the side includes various information about the exercise that will be performed. The first portion 3501 includes information such as the name 3502 of the activity (e.g., Ligating Loop), a [maximum] time duration 3503 for the activity (e.g., 3 minutes), and criteria 3504 indicating a passing score. The criteria 3504, in various embodiments, lists a minimum threshold that would allow for a passing score to be awarded by the surgical performance automated assessment system. Example criteria include how long the activity was performed for and quantifying characteristics of the performance within the activity such as smoothness and accuracy. In addition, the first portion 3501 includes a list of activity instructions 3505 that can detail what cannot be done during the activity lest an invalid attempt be determined. As illustrated in the figure, such instructions may include informing that a hand needs to be kept out of the environment, that the instruments should be maintained within the environment, that the performance of the activity should not exceed a predetermined period of time, and / or that the activity has not been properly finished. In various embodiments, the list of activity instructions can also include instructions on how to perform the activity properly from the beginning to the appropriate end state.Docket No.: 10133-USP-PCT

[0347] In various embodiments, a start button 3506 is provided. When selected, the system starts the selected activity as previously described. The “start” button 3506 may be greyed out or not selectable until confirmation that the instructions and related information for the activity have been read.

[0348] With reference to the second portion (on the left) 3507 of the display, the real-time image of the environment associated with the activity to be performed (e.g., ligating loop) is shown. The real-time image is acquired by the camera of the surgical trainer. In various embodiments, the real-time image displays at least a portion of the surgical exercise within a framed view that allows adequate viewing of the surgical exercise on the display.

[0349] In various embodiments, a full screen option 3513 can be provided. When selected, the full screen option 3513 may minimize or hide the entirety of the first portion 3501. This would allow expansion of the second portion 3507 so that the real-time image of the environment can be expanded and shown over a larger area. Another option (not shown) can also be provided so that the user is capable of bringing back the first portion as needed; for example, if reference to the activity instructions 3505 is needed to be performed at a later time during the activity.

[0350] As shown in FIG. 78F, after the “start” button 3506 has been selected, a notification 3514 can be provided on the display informing a user to start the activity. In various embodiments, the notification may include instructions to insert the instrument(s) to be used with the exercise into the surgical trainer. As noted above, the system needs to be able to detect the instruments being used else the exercise cannot be started and / or be determined to be invalid / failed attempt. In various embodiments, the notification 3514 may include information about what instruments should be used.

[0351] In accordance with various embodiments, prior to the “start” button 3506 being selectable, the system determines the surgical exercise within the camera view matches the expected or predetermine surgical exercise for the selected surgical activity. In various embodiments, for example, prior to the system enabling the “start” button 3506 to be selectable or even provided, the system determines if a ligating loop exercise is positioned within the camera view. If the system determines that the exercise does not match or correspond to the expected or predetermined type of exercise, then the system generates an error, providing a notification, e.g., notification 3523, indicating that a different exercise and / or portions thereof should be used for the particular activity, as shown for example in FIG. 78D. In various embodiments, for example,Docket No.: 10133-USP-PCTthe notification may include instructions to ensure that the particular expected or predetermined surgical exercise of the given or selected surgical activity is being used. As shown in FIGS. 78B-C, similar notifications and / or checks, in various embodiments, are provided or performed by the system such as checking the surgical trainer’s position and / or configuration, e.g., tilt, and / or the presence or removal instruments within the camera. In various embodiments, such checks are initiated via a “setup check” button 351 provided by the system, as shown for example in FIG.78A. In various embodiments, setup instructions and / or notifications also provided to initiate the setup check operations. Once the system determines that the surgical exercise matches the expected or predetermined surgical exercise and / or passes or completes other setup checks, e.g., surgical trainer positioning and / or no visible instrumentation, then the system enables the “start” button, allowing it to be selectable. As shown, a progress portion 3524 can be provided by the system providing information or notifications that setup checks are in progress or being performed by the system, including for example, checking the ligating media, exercise and / or portions thereof presence, placement and / or configuration (e.g., FIG. 78D), instruments are not present and / or removed (e.g., FIG. 78C), and / or the surgical trainer’s position and / or configuration (e.g., FIG.78B).

[0352] Along with the notification 3514 that would be used to direct the user to start the activity, there is also a “finish” button 3522, as shown for example in FIGS. 78F-G. The “finish” button 3522 is selectable after completion of all the steps in the activity. In various embodiments, the “finish” button 3522 may be greyed out until the computing system detects all the objects nearing an end state (e.g., all the instruments are being removed). In various embodiments, the “finish” button 3522 may be greyed out for a pre-determined period of time.

[0353] When selected, the “finish” button 3522 informs the system to stop recording of the performance of the activity. In various embodiments, a prompt may appear that warns the participant that all instruments should be removed from the environment and to ensure that the activity has been fully completed after which another selection of the “finish” button 3522 would terminate the activity. In various embodiments, termination of the activity may automatically be performed a pre-determined period of time after selection of the “finish” button 3522.

[0354] In accordance with various embodiments, exemplary displays are shown in FIGS. 78H-78J configured to capture photos of the ligating loop exercise for analysis, e.g., metrics determinations and / or knot determinations and / or assessments, by the system. In variousDocket No.: 10133-USP-PCTembodiments, such displays or screens are provided after the ligating loop surgical activity is finished or terminated as determined by the system. In various embodiments, a real-time image capture of the environment (e.g., the surgical exercise within the surgical trainer) is displayed as provided by the system. A notification, as generated and displayed by the system, provides instructions to place the appropriate, expected, and / or predetermined ligating loop exercise within a capture frame, e.g., a non-greyed out portion and / or delimited by a greyed our portion, within the real-time image as provided by the system. For example, as shown, in FIG. 781, a notification 3526a, provides that a ligating loop media or exercise should be placed within a square or rectangular capture frame 3527a to take or capture a photo of the ligating loop media, exercise, and / or portions thereof for capture, uploading, detection, determination, and / or analysis by the system. The notification 3526a can provide additional information such as notifying that the ligating loop media should be completely within the capture frame and entirely visible or not covered. A description of instructions 3505 relevant or associated with the other displayed information and / or operation of the surgical activity can also be provided, e.g., providing guidance on how the photo of the expected ligating loop exercise should be performed.

[0355] In various embodiments, a capture button 3525 is provided by the system. When selected, the system captures the image of the ligating loop media, exercise or portions thereof. The capture button 3525 is greyed out or not selectable until the system determines that the appropriate, expected, and / or predetermined ligating loop media, exercise or portions thereof within the capture frame. In various embodiments, the system provides the captured image, e.g., a photo 3528a of the ligating loop media, exercise or portions thereof, and / or a submit button 3529a is provided by the system to initiate the next or finishing step or operation of the surgical activity. In various embodiments, with the submit button selected and detected by the system, the system uploads and / or analyzes the captured photos, e.g., photos of the ligating loop media, exercise or portions thereof. In various embodiments, the system analyzes the photos to determine knot metrics, determinations, and / or score.

[0356] With reference to FIG. 78K, an example feedback screen is shown. The feedback screen is provided after the ‘finish’ selection is selected (see 3315; FIG. 33) during a ligating loop activity. In the example feedback screen 3700, an upload status bar 3701 is shown which provides a status update on how far along the process of uploading the recording of the video data has been completed. The upload status bar 3701 also provides a warning that informs that the video is beingDocket No.: 10133-USP-PCTuploaded and that the current page should not be closed. In various embodiments, a numerical value (e.g., percentage) is provided to inform how far along the upload is at. Furthermore, the upload bar can fill up (from left to right) to correspond with the numerical value shown. In various embodiments, the numerical value may be a countdown timer which estimates how long until the upload has been completed.

[0357] In various embodiments, while the upload is being performed, information about the performance of the activity is provided. In various embodiments, the feedback is user specific. In the feedback screen 3700, a portion shows the criteria for passing the activity being performed. The criteria are all listed within a criteria box 3702 which includes threshold values for one or more individual criteria that would be considered passing. The criteria are specific to the activity; different criteria may be used for different activities. For example, example criteria may include time, smoothness, and / or accuracy. In various embodiments, each of the criteria listed within the criteria box 3702 would need to be at or better the passing level for the activity to have been deemed passing.

[0358] The feedback screen 3700 also includes a current score box 3703. The current score box 3703 includes the current performance and corresponding valuations / scores of the different criteria. The respective scores shown in the current score box 3703 are provided by the system after an evaluation of the user’s performance. In the example, all the criteria have met or performed at a better level than the passing score threshold listed in the criteria box 3702.

[0359] The scores highlighted in the current score box 3703 are calculated based on information determined by the surgical training evaluator regarding each of the criteria specific for the surgical activity. In various embodiments, the surgical training evaluator determines metrics for the surgical activity. In various embodiments, the determined metrics comprise knot tightness and / or knot location or placement accuracy. In various embodiments, the surgical training evaluator assigns the determined metric to a score and / or a pass or fail indication. In various embodiments, the surgical training evaluator assigns the determined metric to a score using a percentage or predetermined scale, e.g., 0 to 100. Furthermore, in various embodiments, the scores are used to calculate a high score shown, for example, in the high score box 3704.

[0360] The feedback screen 3700, in the illustrated embodiment, also includes a high score box 3704. The high score box 3704 shows the one attempt with the highest “total” score. To facilitate the “total” score comparison, criteria (such as time) may need to be assigned a value onDocket No.: 10133-USP-PCTthe 0-100 score value. The best score would be obtained by comparing the scores from all past attempts and selecting the one performance with the highest value. In various embodiments, the high score box 3704 is user specific. In various embodiments the high score box 3704 can be universal (i.e., covers multiple users).

[0361] The feedback screen 3700, in various embodiments, includes a graph 3705 which illustrates performance of one or more criteria over a number of different attempts. For example, as shown in FIG. 78L, the graph 3705 illustrates the difference in time needed to complete the activity by the user over four attempts. Other criteria can also be shown graphically. In various embodiments, the system is capable of displaying each of the criteria based on user selection from the feedback screen 3700. In various embodiments, each of the criteria may be displayed in graphs 3705 altogether (e.g., one on top of each other). The graph 3705 may include a notification 3708 informing the user of how well the user performed at the current attempt compared to the previous attempt.

[0362] Once the upload bar 3701 has completely filled, the upload bar 3701 may be replaced with an upload completion notice informing that the upload has been completed within the feedback window 3700. In various embodiments, the feedback window can include a retry button and / or an exit button. The retry button, in various embodiments, resets the current activity to the beginning. Setup for the activity would need to be performed (and the system confirms the setup is correct), as shown for example, in FIGS. 78A-D, before the activity can be repeated. The exit button, in various embodiments, terminates the current activity and allows selection of a different activity.

[0363] With reference to FIG. 78M, the feedback screen shows an error 4111 associated with the photos being displayed. In various embodiments, the photos are stored in a separate location from the system. Thus, when a photo is requested, photo data is retrieved and then processed by the system to provide the photo in the feedback screen. The notice provided in the error 4111 highlights the possible cause (e.g., the inability to communicate with the storage location over the internet).

[0364] FIG. 79 illustrates an exemplary assessment history page. The assessment history page provides activity specific information for a specific activity. As shown, the assessment history page 4900” is specific to the ligating loop activity. From the assessment history page 4900, the surgical performance automated assessment system provides information about a user’sDocket No.: 10133-USP-PCTperformance related to an activity 4909” and the capability to start the activity 4910. The ligating loop activity 4909 information includes scores related to the highest or best performance 4911 by the user. In various embodiments, there may also be additional notes 4912 that are directed towards improving performance. In various embodiments, the additional notes 4912 may be specific to the user and / or based on past attempt scores. In various embodiments, the additional notes 4912 may be generic suggestions on how to improve one or more criteria scores. With the highest or best performance 4911 by the user, the surgical performance automated assessment system also provides the associated photos 4918 and video recording 4913, allowing, for example, a past performance tied to their best performance on the activity to be viewed. In various embodiments, any past video recording associated with any past attempt can be retrieved and displayed within the assessment history page 4900.

[0365] In various embodiments, a graph 4914 is also provided that shows user attempts over a number of different attempts. The graph 4914 shows one graph for a criterion at a time; with different graphs, in various embodiments, being displayable based on selecting one of the criteria options within the graph 4914. Furthermore, the graph 4914 shows a progression towards the passing score 4916, which in various embodiments is denoted by a dashed line. In various embodiments, the graph 4914 displays a pre-determined number of previous attempts (e.g., last 4 attempts as illustrated in the figure). In various embodiments, any subsequent attempt would remove the oldest attempt from the graph 4914.

[0366] With reference to the assessment history page 4900, information about whose information is tied to the assessment history page 4900 is shown 4908. Thus, the information shown is user-specific and, in various embodiments, would require a login to access. The assessment history page 4900 also provides the ability to go to other menus associated with other features or functions of the surgical performance automated assessment system. “Home” 4901 brings back up the main page. “Courses” 4902 brings up the different activities and / or difficulties that can be selected. “My videos” 4903 links or provides past recorded and uploaded video data of past attempts on all the activities associated with the surgical performance automated assessment system. In various embodiments, “Video Library” 4904 provides access to all past recorded and uploaded video data of the other users. “Assessment History” (as discussed above) 4905 provides information about past performances and comparison between the past performances. In variousDocket No.: 10133-USP-PCTembodiments, “Resources” 4907 may include informational videos that can be viewed to teach how to perform an activity and / or skills associated with the activity.

[0367] In accordance with various embodiments, one or more of the surgical exercises 40 comprises a sleeve transfer surgical exercise 40h. The sleeve transfer surgical exercise comprises a plurality of peg boards or skin pads 402a-e within a substantially circular bowl 81. The bowl 81 has a plurality of angled side walls extending away from a flat or planar center base. As illustrated, the skin pads have varying shapes and sizes with different patterns of holes with each hole configured to receive a post or peg. For example, a first skin pad 402a is positioned on an angled side wall and has a plurality of holes arranged in a pattern, e.g., a three-by-three square pattern, and each of the plurality of holes are configured to receive a peg or post. An adjacent second skin pad 402b is positioned on an adjacent angled side wall of the bowl 81 and has a different shape, i.e., trapezoidal, then the shape of the first skin pad 402a, e g., rectangular. The pattern and / or number of holes in the second skin pad 402b is also different than the pattern and / or number of holes in the first skin pad 402a. In various embodiments, each skin pad may have no posts extending therefrom or one or more posts extending therefrom. Each post is configured to receive a ring or sleeve and one or more of the plurality of posts extend in an orthogonal direction relative to their respective peg board or skin pad. Each of the plurality of sleeves are movable relative to each of the plurality of posts and the peg board and / or one or more sleeves are movable from one post to another post. The sleeve transfer surgical exercise further comprises a center and / or flat peg board 420 having one or more posts 421 extending orthogonally therefrom. In various embodiments, the posts of the center peg board are positioned in a pattern, e.g., a three-by-three rectangle. Each post is configured to removably receive a ring or sleeve 422. In various embodiments, a sleeve on a post on the center peg board 420 is moveable to a post 401a-f on one of the angled skin pads 402a-e. In various embodiments, a sleeve positioned on a post on one of the angled skin pads 402a-e is movable to a post on the center peg board 420.

[0368] Each of the plurality of sleeves has an opening sized, shaped and / or dimensioned to receive a post therethrough. In various embodiments, each of the plurality of sleeves share or have a common or equivalent shape, size, dimension, and / or color to each other. In various embodiments, each of the plurality of sleeves has a different or does not have or share a common or equivalent shape, size, dimension, and / or color compared to the center peg board and / or the plurality of angled skin pads. In various embodiments, the angled skin pads comprise a base boardDocket No.: 10133-USP-PCThaving a plurality of holes with a simulated skin layer disposed over an upper layer of the base board. The simulated skin layer has a plurality of holes aligned with the plurality of holes of the base board. Each of the plurality of holes of the simulated skin layer and each of the plurality of holes of the base board are configured to receive a peg or post. In various embodiments, the angled skin pads comprise a simulated skin layer with a plurality of holes configured to receive a peg or post and the simulated skin layer directly attached to an angled side wall of the bowl 81. In various embodiments, the simulated skin layer and / or base board are flat or planar with the angled side wall of the bowl providing the angle or slant of the simulated skin layer, base board, and / or skin pad relative to the flat or planar center of the bowl and / or the center peg board. In various embodiments, the center peg board does not have a simulated skin layer and comprises a base board with a plurality of holes configured to receive a peg or post.

[0369] In various embodiments, the sleeve transfer surgical exercise is configured to be removably connectable or able to be positioned directly on top of the base 14 of the surgical trainer and is configured to be positioned between the cover 11 and the base 14 of the surgical trainer and / or within a cavity defined by the cover 11 and base 14 of the surgical trainer. In various embodiments, the bowl 81 of the sleeve transfer surgical exercise comprises a hook and loop type interface to removably secure the surgical exercise to the base of the surgical trainer and / or is otherwise prevented or restricted from movement, e.g., X, Y, or Z axis movement, relative to the base of the surgical trainer.

[0370] In various embodiments, the sleeve transfer surgical exercise is configured to ensure the view or camera image of the surgical exercise is optimal, e.g., able to be analyzed and / or evaluated by the surgical training evaluator. Given the dimensions of the sleeve transfer surgical exercise, the surgical training evaluator is configured to determine the surgical exercise’s placement and / or the tilt of the surgical trainer using segmentation and / or a Perspective-n-Lines (PnL) algorithm. In various embodiments, the surgical evaluator is configured to determine the 3D position and / or orientation of the surgical exercise relative to the camera. In various embodiments, the surgical training evaluator uses the PnL algorithm to determine a 3D position and / or orientation of a surgical exercise, e.g., the sleeve transfer surgical exercise, relative to the camera. In various embodiments, the surgical evaluator, utilizing the PnL algorithm, determines the 3D position and 3D orientation / rotation of the sleeve transfer surgical exercise relative to the camera. The surgical evaluator, utilizing the determined 3D rotation, determines the tilt or angleDocket No.: 10133-USP-PCTof the sleeve transfer surgical exercise relative to the camera, which is equivalent to the tilt of the surgical trainer. The surgical evaluator utilizing the PnL algorithm generates and / or places a 3D model of the surgical exercise in a 3D space, and projects how the surgical exercise is viewed by the camera. The surgical training evaluator detects 2D lines in the camera image of the sleeve transfer surgical exercise within the camera view and adjusts or moves the generated 3D model to overlap the detected 2D lines.

[0371] In various embodiments, the sleeve transfer exercise comprises a plurality of skin pads, e.g., five skin pads 402a-e, which are distinct from the bowl 81. In the illustrated embodiment, the color of the skin pads 402a-e is different and distinct from the bowl 81 and using segmentation, the surgical evaluator identifies the skin pads as shown in FIG. 87A and represented by masks 406a-e. With the five pads are identified, the surgical evaluator is configured to identify and match the sides or partial outlines 408a-e of the plurality of skin pads to the generated 3D model of the sleeve transfer exercise using PnL, as shown for example in FIG. 87B. As such, the surgical evaluator determines the placement and alignment of the generated 3D model with the camera view in which the generated 3D model aligns with the camera view of the sleeve transfer surgical exercise providing the surgical evaluator the sleeve transfer surgical exercise. As shown for example in FIG. 87C, the lines or outline 405a-e illustrate the best fit of the 3D model overlayed with the 2D lines or partial outlines 408a-e identified by the surgical evaluator. As shown in FIG.87C, hole locations of the skin pads are also identified, a subset of which is labeled 409a-e for illustrative purposes. Utilizing this information, the surgical evaluator is configured to measure the distance between where the surgical exercise is placed and the desired target location of the surgical exercise.

[0372] In various embodiments, the surgical evaluator is configured to guide the user to position the surgical exercise in an optimal target location in which the surgical evaluator can analyze and / or assess the surgical activity performed and the state of the surgical exercise. In the illustrated embodiment shown in FIGS. 83-84, the surgical evaluator is configured to display a red overlay 405a-e on the screen indicating the target location of the sleeve transfer surgical exercise. When the sleeve transfer surgical exercise is positioned to match the red overlay, the surgical evaluator changes the color of the overlay, e.g., from red to green, as shown for example in FIG.86.Docket No.: 10133-USP-PCT

[0373] In various embodiments, the surgical evaluator determines and / or performs setup checks of the sleeve transfer surgical exercise, e.g., similar to the setup checks of other surgical exercises. In various embodiments, using segmentation and ROIs, the surgical evaluator is configured to determine that the plurality of posts are placed in a predetermined holes in the skin pad. In various embodiments, the surgical evaluator is configured to determine that all the posts in the center ped board are present and placed in predetermined holes in the center peg board. In various embodiments, the surgical evaluator is configured to determine that a sleeve is on each of the posts of the center peg board. In various embodiments, posts, sleeves, skin pads and / or peg board, checks provided by the surgical evaluator uses segmentation and the identified location of the sleeve transfer exercise to validate that the sleeve transfer exercise is in a predetermined placement, e.g., its optimal target location. In various embodiments, the surgical evaluator determines the tilt or angle of the surgical trainer or that the surgical trainer is tilted or angled, e.g., its cover is angled or slanted relative to its base, as part of the setup check or validation of the sleeve transfer surgical exercise.

[0374] In accordance with various embodiments, the surgical training evaluator utilizing semantic segmentation, contour determination, and / or ROIs determines if one or more posts are present or absent on each of the skin pads and the center peg board, the position of each of the posts relative to a corresponding skin pad and the center peg board, and / or the position of the sleeves on the center peg board matches a predetermined number of pegs, peg pattern or position, and / or the sleeve positions or pattern for each of the skin pads and the center peg board. If the surgical training evaluator identifies a failed match or comparison, the surgical training evaluator, in various embodiments, provides an error notification noting the general and / or specific failure, e.g., the position of the missing post or sleeve. In various embodiments, the surgical training evaluator provides steps, instructions, and / or hints on how to correct the issue.

[0375] In various embodiments, the sleeve transfer activity utilizes predetermined ports 18a-c of the surgical trainer, e.g., either ports 2 (18a) and 11 (18c), or ports 11 (18c) and 3 (18b), as shown in FIG. 88. In other embodiments, different ports and / or different number of ports may be predetermined and / or utilized for the sleeve transfer activity. As shown in FIG. 89A, the illustrated diagram represents what the camera views in which the camera is located at predetermined point A and the rectangle BCDE represents the camera image or view. The yellow line segment JH is the surgical instrument viewed from the camera and in various embodiments, the start point J andDocket No.: 10133-USP-PCTend point H of the surgical instrument in the camera image are predetermined. As shown, the cylinder FG is the surgical instrument in 3D space, and the black circle F is the port. In various embodiments, the surgical evaluator is configured to determine point F, where the surgical instrument intersects with the port of surgical trainer, by utilizing the predetermined camera position A, and the start and end of the surgical instrument in the image, J and H.

[0376] By taking two measurements with the surgical instrument on opposite sides of the screen or display, i.e., an instrument entering from the left and pointed to the right (e.g., as shown in FIG. 89B) and an instrument entering from the right and pointed to the left (e.g., as shown in FIG. 89C), the surgical evaluator determines a relative angle to find the pivot point and / or port location. The surgical evaluator utilizes the three known points A, H, and J, defining a plane. With two separate measurements, one from the right, i.e., the instrument pointed to the right, and one from the left, i.e., the instrument point to the left, the two planes, a right plane 182 defined by points A, H, and J, e.g., as shown in FIG. 89D, and a left plane 183 defined by points A, H, and J, e.g., as shown in FIG. 89E, intersect each other, as shown in FIG. 89F.

[0377] By utilizing the intersection of these two planes, the surgical evaluator determines or identifies a line at the intersection of the planes, e.g., line 181, as shown in 89F. The pivot point or port utilized for the surgical instrument must lie somewhere on this line 181. With the predetermined position of the camera inside the trainer, the surgical evaluator is configured to determine the intersection of this line with the top plane of the surgical trainer to determine the pivot point of the surgical instrument.

[0378] The surgical evaluator compares the predetermined locations of ports, e.g., ports 2, 3 and 11 for an assigned or specific surgical exercise, e.g., a sleeve transfer surgical exercise, to the determined pivot point location of the surgical instrument to determine which port is used, i.e., the entry or insertion port of the surgical instrument inserted into the surgical trainer. If the pivot point location substantially matches the predetermined location of port 2, the surgical evaluator determines the port being used as port 2. If the pivot point location substantially matches the predetermined location of port 3, the surgical evaluator determines the port being used as port 3. If the pivot point location substantially matches the predetermined location of port 11, the surgical evaluator determines the port being used as port 11. Any mismatch determination by the surgical evaluator causes the surgical evaluator to provide or display a notification of the mismatch and / or steps to correct the mismatch, the use of the incorrect port and what port should be used. If noDocket No.: 10133-USP-PCTmismatches or at least two matches are determined by the surgical evaluator, the surgical evaluator provides or displays a notification that calibration is complete and / or the surgical activity can be started. As such, the surgical evaluator is configured to determine if a predefined pivot point or port is being utilized based on the selected or assigned surgical activity and / or exercise. The above specific and / or number of port / pivot point or ports / pivot points predefined, determined, and verified may vary based on the surgical activity and / or exercises and / or other conditions including changes to the level of difficulty or ease desired for the surgical activity, the assessments being made, and / or the surgical skill or task being trained, practiced, and / or tested. For example, in one embodiment, for a particular surgical exercise, e.g., the sleeve transfer surgical exercise, may have ports 2, 3, and 11 predefined, determined and verified by the surgical evaluator and in another embodiments, the sleeve transfer surgical exercise may have ports 2, 3 and 14 predefined, determined and verified by the surgical evaluator. In another embodiment, for example, the sleeve transfer surgical exercise or the bifurcation surgical exercise may have only two ports, e.g., ports 2 and 11, predefined, determined and verified by the surgical evaluator.

[0379] To determine the pivot point of the surgical instrument and / or the port being used, in various embodiments, the surgical evaluator is configured to provide a calibration process. In various embodiments, the surgical evaluator is configured to require the calibration process to complete before each attempt of the sleeve transfer surgical exercise to verify that the ports are as predetermined or correct. The surgical evaluator as such provides or displays a highlighted area, e g., left circle 412. Both surgical instruments 30a, b are required to be moved to a left circle 412 on the left (e.g., as shown in FIG. 90B), upon which the surgical evaluator is configured to perform the first calibration measurement, i.e., the first measurement previously described in reference to FIGS. 89C,E. The surgical evaluator also provides or displays a highlighted area, e.g., right circle 414. Both surgical instruments 30a, b are required to be moved to a right circle 412 on the right (e.g., as shown in FIG. 90D), upon which the surgical evaluator is configured to perform the second calibration measurement, i.e., the second measurement previously described in reference to FIGS.89B,D. Utilizing the previously described measurements, calculations, and comparisons, if the surgical evaluator determines that incorrect ports, e.g., not a predefined port or ports, e.g., for or assigned for a particular or selected surgical activity, are used, then the activity is aborted, and the surgical evaluator provides prompts to remove the surgical instruments and use the correct ports,Docket No.: 10133-USP-PCTat which point the calibration is performed again by the surgical evaluator. Once calibration is successful then the surgical evaluator provides a start circle 415 to initiate the surgical activity.

[0380] In various embodiments, once the surgical activity begins or is initiated, the surgical evaluator tracks the surgical instruments to generate metrics and / or that each of the sleeves are moved from each of the posts of the center peg board to an outer post on the outer skin pads and then back to the posts of the center peg board. In various embodiments, the surgical training evaluator is configured to determine and / or set a plurality of regions of interest (ROIs) overlaying the pegs on each of the skin pads and the center peg board. In various embodiments, the surgical training evaluator utilizing identified ROIs at each of the respective posts is configured to identify when a sleeve is positioned from one post to another post. As such, the surgical evaluator in realtime is configured to track the progress of the placements of the sleeves and, in various embodiments, provide progress updates to a user on the activity, e.g., the correct or incorrect placement of sleeves and / or remaining locations or number of sleeves. In various embodiments, the progress details can also be recorded, stored, and presented later, e.g., via a feedback screen. In particular, for a sleeve transfer activity, each of the six sleeves are moved from the posts one at time from the center peg board to one of the unoccupied posts on the outer skin pad, and then back from those posts to each of the posts on the center peg board. Accordingly, utilizing identified ROIs, as noted above, the surgical training evaluator is configured to track the progress of the activity. In various embodiments, the surgical evaluator is configured to determine surgical instrument metrics (e.g., path length, smoothness...) and are recorded and / or stored, along with the time needed to complete the sleeve transfer surgical activity. In various embodiments, the surgical instrument metrics, completion time, and / or other assessment metrics are recorded, stored, and presented in real time, e.g., while a user is performing a surgical activity, after a user completes an attempt to perform the surgical activity, and / or once a metric or other assessment information is determined by the surgical training evaluator.

[0381] It should be noted that in various drawings illustrating exemplary regions of interest, areas of interest, masks, contours, and the like are provided for illustrative purposes and to ease description of operations performed by the assessment system and / or the surgical training evaluator and are not shown or displayed on a screen or monitor to a user or the like.

[0382] In accordance with various embodiments, the surgical performance automated assessment system and, in various embodiments, the surgical training evaluator, e.g., comprisingDocket No.: 10133-USP-PCTone or more processors configured to execute machine-readable instructions, provides the abovedescribed displays, notifications, and / or associated information therein including the determination, processing, identification, and / or interaction therewith to provide the associated displays, notifications, and / or the associated information therein utilizing for example one or more processors.

[0383] The above description is provided to enable any person skilled in the art to make and use the devices or systems and perform the methods described herein and sets forth the best modes contemplated by the inventors of carrying out their inventions. Various modifications, however, will remain apparent to those skilled in the art. It is contemplated that these modifications are within the scope of the present disclosure. Different embodiments or aspects of such embodiments may be shown in various figures and described throughout the specification. However, it should be noted that although shown or described separately each embodiment and aspects thereof may be combined with one or more of the other embodiments and aspects thereof unless expressly stated otherwise. It is merely for easing readability of the specification that each combination is not expressly set forth.

[0384] Although the present invention has been described in certain specific aspects, many additional modifications and variations would be apparent to those skilled in the art. It is therefore to be understood that the present invention may be practiced otherwise than specifically described, including various changes in the size, shape and materials, without departing from the scope and spirit of the present invention. Thus, embodiments of the present invention should be considered in all respects as illustrative and not restrictive.

Claims

Docket No.: 10133-USP-PCTClaims:

1. A surgical performance automated assessment system comprising:a surgical trainer comprising a camera positioned to capture video data of a surgical exercise positioned within the surgical trainer; anda surgical training evaluator comprising one or more processors configured to analyze the captured video data in which the one or more processors are configured to:convert pixels from the captured video data to a YUV color space in which each converted pixel is assigned a brightness component (Y), a first color projection component (U), and a second color projection component (V);identify one or more areas of interest (AOIs) in the converted pixels within a UV color space from the YUV color space, the UV color space comprising the first and second color projection components of the converted pixels and ignoring the brightness component, each AOI of the identified one or more AOIs corresponding to a predefined object of interest;generate a mask designating pixels in a frame of the captured video data that correspond to an AOI of the identified one or more AOIs; anddetermine a state of a surgical activity for the surgical exercise or recognizing the predefined object of interest based on the generated mask being located within a predefined region of interest in the frame of the captured video data.

2. The system of claim 1, wherein the one or more processors of the surgical training evaluator are configured to determine the state of the surgical activity for the surgical exercise based on the generated mask being located within the predefined region of interest in subsequent frames of the captured video data.

3. The system of claims 1 or 2, wherein the one or more processors of the surgical training evaluator are configured to recognize the predefined object of interest based on the generated mask being located the frame of the captured video data.Docket No.: 10133-USP-PCT4. The system of claims 1, 2 or 3, wherein the one or more processors of the surgical training evaluator are configured to track locations of the predefined object of interest based on locations of the generated mask in subsequent frames of the captured video data.

5. The system of claim 4, wherein the one or more processors of the surgical training evaluator are configured to calculate a metric based on the track locations of the predefined object of interest.

6. The system of any one of the previous claims wherein the surgical trainer comprises a cover and a base; a camera connected to a lower surface of the cover of the surgical trainer; a light connected to the lower surface of the cover of the surgical trainer; and an exercise insert adapter removably connected to the base of the surgical trainer, the surgical exercise being removably connected to the exercise insert adapter.

7. The system of claim 6 wherein the camera is fixed to the cover of the surgical trainer, not movable or adjustable, and angled relative to the base and the cover of the surgical trainer.

8. The system of any one of the previous claims wherein the light is fixed to the cover of the surgical trainer, adjacent to the camera, and facing away from the camera.

9. The system of any one of the previous claims wherein the cover is configured to tilt relative to the base via a plurality of legs connecting to the base to the cover of the surgical trainer.

10. The system of any one of the previous claims wherein the cover comprises a removable plate comprising a plurality of pre-arranged access ports through which a surgical instrument is insertable therethrough to access the surgical exercise.

11. The system of any one of the previous claims wherein the surgical exercise comprises at least one of a peg board surgical exercise, precision cutting surgical exercise, tissue dissection surgical exercise, ligating loop surgical exercise, a sleeve transfer surgical exercise or a petal surgical exercise.Docket No.: 10133-USP-PCT12. The system of claim 11 wherein the peg board surgical exercise, the precision cutting surgical exercise, the tissue dissection exercise, and the ligating loop exercise comprises a platform removably connected to the exercise insert adapter.

13. The system of claim 12 wherein the platform of the peg board surgical exercise is a base board and the task trainer comprises a plurality of rings and plurality of posts, the plurality of posts fixedly connected to the base board and orthogonally extending away from the base board, each ring of the plurality of rings removably connected and movable from one post of the plurality of posts to another post of the plurality of posts.

14. The system of claim 13 wherein the plurality of rings comprises a first set of rings and a second set of rings with each ring of the first set of rings having a first common characteristic to each other and each ring of the second set of rings having a second common characteristic, the second common characteristic being different from the first common characteristic.

15. The system of claim 11 wherein the surgical exercise comprises a first peg board surgical exercise and a second peg board surgical exercise, the second peg board surgical exercise being different from the first peg board surgical exercise.

16. The system of claim 15 wherein the surgical training evaluator configured to distinguish the first peg board surgical exercise from the second peg board surgical exercise.

17. The system of claim 12 wherein the platform of the petal surgical exercise is a petal base board and the task trainer comprises a central mass, a stem, and a plurality of arms, the plurality of arms extending away from the central mass along a first axis and a stem extending away from the central mass along a second axis, the first axis being orthogonal to the second axis; wherein the central mass and the plurality of arms are removably connected to the petal base board.Docket No.: 10133-USP-PCT18. The system of claim 12 wherein the platform of the precision cutting surgical exercise is a cut base and the task trainer comprises cutting media including a cut pattern, the cutting media removably connected to the cut base.

19. The system of claim 18 wherein the cut base comprises a clip, the cutting media being removably connected to the clip of the cut base.

20. The system of claims 18 or 19 wherein the cut pattern delimits the cutting media into two separable portions, a first inner portion and a second outer portion.

21. The system of claim 12 wherein the platform of the ligating loop surgical exercise is a ligation base, and the task trainer comprises ligation media including a plurality of fingers, the ligation media removably connected to the base.

22. The system of claim 21 wherein the ligation base comprises a clip extending along an upper surface and a distal portion of the ligation base, the ligation media being removably connected to the clip of the base.

23. The system of claim 12 wherein the platform of the dissection surgical exercise is a base, and the task trainer is a simulated tissue structure comprising of one or more bifurcations.

24. The system of claim 12 wherein the sleeve transfer surgical exercise comprises a bowl, a plurality of skin pads, a plurality of sleeves, and a peg board, one or more posts connected to each skin pad of the plurality of skin pads and extending orthogonally away from each skin pad, a plurality of posts connected to the peg board and extending orthogonally away from the peg board, each sleeve of the plurality of sleeves removably connected and movable from one post of the plurality of posts of the peg board to the one or more posts of each skin pad of the plurality of skin pads.

25. The system of any one of the previous claims wherein the base of the surgical trainer comprises front edges and back edges and the exercise insert adapter comprises front edgesDocket No.: 10133-USP-PCTengageable with the front edges of the base of the surgical trainer and the exercise insert adapter comprises back edges engageable with back edges of the base of the surgical trainer.

26. The system of any one of the previous claims wherein the exercise insert adapter engages with the base of the surgical trainer to prevent movement of the exercise insert adapter and the surgical exercise along multiple axes relative to the base and operational manipulation by a user of the task trainer of the surgical exercise.

27. The system of any one of the previous claims wherein the exercise insert adapter comprises a front flange and a back flange engageable with outer edges of the platform of the surgical exercise and the exercise insert adapter comprises side tabs engageable with inner edges of the platform of the surgical exercise to prevent movement of the surgical exercise relative to the exercise insert adapter and operational manipulation by a user of the task trainer of the surgical exercise.

28. The system of any one of the previous claims wherein the exercise insert adapter, or the surgical exercise comprises a tilt indicator, the tilt indicator being the predefine object of interest.

29. The system of any one of the previous claims wherein the surgical instrument comprises an elongate shaft comprising a sleeve being the predefined object of interest.

30. A surgical performance automated assessment system comprising:a surgical training evaluator comprising one or more processors configured to analyze video data in which the one or more processors are configured to:convert pixels from the video data to a YUV color space in which each converted pixel is assigned a brightness component (Y), a first color projection component (U), and a second color projection component (V);identify one or more areas of interest (AOIs) in the converted pixels within a UV color space from the YUV color space, the UV color space comprising the first and second color projection components of the converted pixels, each AOI of the identified one or more AOIs corresponding to a predefined object of interest;Docket No.: 10133-USP-PCTgenerate a mask designating pixels in a frame of the video data that correspond to an AOI of the identified one or more AOIs; anddetermine a state of a surgical activity for the surgical exercise or recognizing the predefined object of interest based on the generated mask being located within a predefined region of interest in the frame of the video data.

31. The system of any one of the previous claims wherein the surgical training evaluator using segmentation is further configured to:identify and isolate a surgical instrument from the video data;generate a mask of the identified and isolated surgical instrument;generate a contour of the generated mask;identify two longest sides of the generated contour and a line parallel to and midway between the two longest sides to determine a two-dimensional direction of the surgical instrument;determining an intersection of the line parallel to and midway between the two longest sides with the generated contour to determine a location of a distal end of the surgical instrument.

32. The system of any one of the previous claims wherein the surgical training evaluator using segmentation is further configured to:identify and isolate a tilt indicator from the video data;generate a mask of the identified and isolated tilt indicator;generate a contour of the generated mask;determine a location of the tilt indicator from the generated contour;comparing the location of the tilt indicator to a lookup table comprising predetermined locations representing predetermined angles of the cover of the surgical trainer relative to the base of the surgical trainer; anddetermining a tilt of the cover of the surgical trainer relative to the base of the surgical trainer based on a result of the comparison.

33. The system of any one of the previous claims wherein the surgical training evaluator using segmentation is further configured to:Docket No.: 10133-USP-PCTidentify and isolate a surgical instrument from the video data;generate a mask of the identified and isolated surgical instrument;generate a contour of the generated mask;identify two longest sides of the generated contour and a line parallel to and midway between the two longest sides to determine a two-dimensional direction of the surgical instrument;determine the two-dimensional direction of the surgical instrument pointing from a left to a right relative to a view of the camera to identify the surgical instrument as a left surgical instrument; anddetermine the two-dimensional direction of the surgical instrument pointing from the right to the left relative to the view of the camera to identify the surgical instrument as a right surgical instrument.

34. The system of any one of the previous claims wherein the surgical training evaluator using segmentation is further configured to determine a ratio of a first pixel color to a second pixel color to identify a first peg board of a surgical exercise is present within the view of a camera when the ratio is greater than one and to identify a second peg board of the surgical exercise is present within a view of the camera when the ratio is less than one.

35. The system of any one of the previous claims wherein the surgical training evaluator is configured to monitor predetermined regions of interest of the surgical exercise to track the progress or state of the surgical activity.

Images