Integration of temporal and spatial features into machine learning assessment of open surgical skills
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2026-02-10
- Publication Date
- 2026-08-13
Smart Images
Figure US2026014713_13082026_PF_FP_ABST
Abstract
Description
Integration of Temporal and Spatial features into Machine Learning Assessment of Open Surgical SkillsFIELD OF THE INVENTION
[0001] The present invention relates generally to surgical education and more specifically to training machine learning models for assessment of surgical skills.BACKGROUND OF THE INVENTION
[0002] The challenge of objectively assessing surgical trainee performance during open procedures has been a persistent concern in medical education. While traditional evaluation methods depend on expert surgeon observation and grading, this approach introduces inherent variability among evaluators. Such inconsistencies not only affect the quality of training feedback but also create significant challenges in standardizing surgical education across institutions and monitoring trainee development longitudinally. As healthcare systems worldwide strive to maintain rigorous standards of surgical competency and patient safety, the necessity for objective assessment tools has become increasingly apparent.SUMMARY OF THE INVENTION
[0003] In one embodiment, a system for automated surgical skill assessment is provided. The system includes a spatial feature extraction module configured to receive video frames captured from a surgical procedure and extract spatial features from the video frames using a convolutional neural network architecture that has undergone domain-specific fine-tuning for surgical video analysis. The system further includes a temporal feature extraction module configured to receive the spatial features from the spatial feature extraction module and analyze sequential relationships between the video frames using a recurrent neural network architecture. An attention mechanism is integrated within the temporal feature extraction module and configured to dynamically assign importance weights to different segments of the surgical procedure. The system also includes a classification module configured to receive temporal features from the temporal feature extraction module and generate a probability distribution across a plurality of skill categories indicating a skill level classification for the surgical procedure.
[0004] Tn other embodiments, the system may include one or more of the following features. The convolutional neural network architecture may comprise a ResNet50 backbone with a custom classification head incorporating Global Average Pooling and dense layers with L1-L2 regularization. The domain-specific fine-tuning may comprise a two-phase training strategy in which the ResNet50 backbone is frozen during a first phase and unfrozen during a second phase. The recurrent neural network architecture may comprise a bidirectional Long Short-Term Memory network configured to process the spatial features in both forward and reverse temporal directions. The attention mechanism may comprise a single-head soft attention mechanism configured to identify technical moments including suture placements and tissue handling while maintaining awareness of overall procedural flow. The plurality of skill categories may comprise a novice skill category, an intermediate skill category, and an expert skill category. The skill categories may be defined based on a Global Rating Score derived from Objective Structured Assessment of Technical Skills evaluation, wherein the novice skill category corresponds to scores less than 16 points, the intermediate skill category corresponds to scores greater than or equal to 16 points and less than 24 points, and the expert skill category corresponds to scores greater than or equal to 24 points. The system may further comprise a feature caching module configured to store extracted spatial features from the spatial feature extraction module to eliminate redundant computation when analyzing previously processed video frames. The classification module may comprise fully connected layers configured to receive the temporal features and transform learned representations into a format suitable for classification, and a softmax activation function configured to generate the probability distribution across the plurality of skill categories. The video frames may be captured from a bird's-eye view perspective at a frame rate of approximately 30 frames per second. The attention mechanism may generate attention weight distributions that differentiate between skill levels by identifying temporal segments where technique differences manifest, wherein novice attention patterns exhibit extended preparation time and hesitation points, and wherein expert attention patterns exhibit efficient preparation and consistent rhythm.
[0005] In another embodiment, a method for automated assessment of surgical skills is provided. The method includes receiving a sequence of video frames captured from a surgical procedure. The method further includes extracting spatial features from the video frames using a convolutional neural network that has been fine-tuned for surgical video analysis. The method includes processing the spatial features through a temporal feature extraction module comprisinga recurrent neural network to analyze sequential relationships between the video frames. The method also includes applying an attention mechanism to dynamically assign importance weights to different segments of the surgical procedure. The method further includes generating a probability distribution across a plurality of skill categories based on the temporal features to classify a skill level of the surgical procedure.
[0006] In other embodiments, the method may include one or more of the following features. The convolutional neural network may comprise a ResNet50 backbone with a custom classification head incorporating Global Average Pooling and dense layers with L1-L2 regularization. Fine-tuning the convolutional neural network may comprise a two-phase training strategy in which the ResNet50 backbone is frozen during a first phase and unfrozen during a second phase. The recurrent neural network may comprise a bidirectional Long Short-Term Memory network configured to process the spatial features in both forward and reverse temporal directions. Applying the attention mechanism may comprise applying a single-head soft attention mechanism configured to identify technical moments including suture placements and tissue handling while maintaining awareness of overall procedural flow. The plurality of skill categories may comprise a novice skill category, an intermediate skill category, and an expert skill category defined based on a Global Rating Score derived from Objective Structured Assessment of Technical Skills evaluation. The method may further comprise caching extracted spatial features to eliminate redundant computation when analyzing previously processed video frames. Applying the attention mechanism may generate attention weight distributions that differentiate between skill levels by identifying temporal segments where technique differences manifest, wherein novice attention patterns exhibit extended preparation time, multiple tool adjustments, and hesitation points, and wherein expert attention patterns exhibit efficient preparation, streamlined execution, and consistent rhythm.
[0007] In yet another embodiment, a non-transitory computer-readable medium storing instructions is provided. The instructions, when executed by a processor, cause the processor to perform operations comprising receiving video frames captured from a surgical procedure. The operations further comprise extracting spatial features from the video frames using a convolutional neural network architecture fine-tuned for surgical technique recognition. The operations comprise analyzing sequential relationships between the video frames by processing the spatial features through a bidirectional recurrent neural network. The operations also comprise applying anattention mechanism to assign importance weights to segments of the surgical procedure. The operations further comprise generating a skill level classification for the surgical procedure based on a probability distribution across skill categories.
[0008] In other embodiments, the non-transitory computer-readable medium may include one or more of the following features. The convolutional neural network architecture may comprise a ResNet50 backbone with a custom classification head incorporating Global Average Pooling and dense layers with L1-L2 regularization. The operations may further comprise training the convolutional neural network architecture using a two-phase training strategy in which the ResNet50 backbone is frozen during a first phase and unfrozen during a second phase. The skill categories may comprise a novice skill category corresponding to a Global Rating Score less than 16 points, an intermediate skill category corresponding to a Global Rating Score greater than or equal to 16 points and less than 24 points, and an expert skill category corresponding to a Global Rating Score greater than or equal to 24 points. The attention mechanism may comprise a singlehead soft attention mechanism that generates attention weight distributions identifying temporal segments where technique differences manifest between skill levels. The operations may further comprise caching extracted spatial features to eliminate redundant computation when analyzing previously processed video frames, to be completed when claims are finalized.BRIEF DESCRIPTION OF THE FIGURES
[0009] FIG. 1 depicts an instrumented surgical tool with an attached electronic circuit board, according to an embodiment of the invention.
[0010] FIG. 2A depicts a video frame from a surgical procedure with hand and instrument detection overlays, according to an embodiment of the invention.
[0011] FIG. 2B depicts a schematic representation of a video frame analysis display of FIG.2A, according to an embodiment of the invention.
[0012] FIG. 3 depicts a data distribution visualization showing categorization of surgical skill assessment scores, according to an embodiment of the invention.
[0013] FIG. 4 depicts a machine learning pipeline for automated surgical skill assessment, according to an embodiment of the invention.
[0014] FIG. 5 depicts a Receiver Operating Characteristic curve analysis demonstrating classification performance, according to an embodiment of the invention.DETAILED DISCLOSURE OF THE INVENTION
[0015] Systems and methods for automated assessment of surgical skills are disclosed, and more particularly video-based machine learning approaches for evaluating open surgical techniques. Objective assessment of surgical trainee performance during open procedures presents challenges in medical education. Traditional evaluation methods depend on expert surgeon observation and grading, which may introduce variability among evaluators. Such inconsistencies may affect the quality of training feedback and may create challenges in standardizing surgical education across institutions and monitoring trainee development over time.
[0016] Embodiments of the invention as described herein may address these challenges through a two-phase deep learning architecture configured to automate surgical skill assessment. The first phase may employ transfer learning through a modified convolutional neural network architecture that undergoes domain-specific fine-tuning to recognize nuances in surgical techniques. Surgical movements have spatial characteristics that differ from general object recognition tasks, and the fine-tuning process can enable the model to learn surgery-specific feature representations.
[0017] The second phase may provide temporal understanding of surgical procedures through a spatiotemporal classification architecture. The architecture may integrate recurrent neural network components with an attention mechanism that mirrors the cognitive process of expert surgeons who evaluate procedures. The attention mechanism may dynamically assign importance to different segments of a procedure, allowing the model to identify technical moments such as suture placements or tissue handling while maintaining awareness of overall procedural flow. The recurrent components may enable the model to understand both precedent and subsequent context of each surgical movement.
[0018] Open surgery presents challenges due to unrestricted movement and variable viewing angles compared to minimally invasive surgery, which benefits from standardized camera angles and controlled environments. Embodiments of the invention may overcome these limitations through the attention mechanism, which may adapt to various perspectives and focusing distances while identifying technical elements. The attention-based approach may also provide consistent, objective assessment across different evaluators and institutions.
[0019] Surgical skill assessment systems according to many embodiments of the invention may understand the hierarchical and progressive nature of surgical skill development. Traditional assessment methods may struggle to capture nuanced transitions between skill levels, particularly in intermediate ranges where differences become more subtle. The systems described herein may demonstrate capability in understanding gradients of skill development through sophisticated feature extraction and temporal analysis capabilities. The model may maintain logical consistency in assessments, distinguishing between novice and expert performances while showing nuanced discrimination between adjacent skill levels.Instrumented Surgical Tools for Data Collection
[0020] Referring to FIG. 1, an instrumented surgical tool according to some embodiments of the invention may be configured for data collection during surgical training and assessment procedures. The instrumented surgical tool can include a hemostat or needle holder forceps having a scissor-like configuration with two finger rings positioned at a proximal end and elongated serrated jaws positioned at a distal end. A locking ratchet mechanism may be disposed between the finger rings and the jaws to enable controlled gripping and release of surgical materials such as suture needles.
[0021] An electronic circuit board may be attached to the instrumented surgical tool. The electronic circuit board may include a sensor module or data acquisition device configured to detect and record instrument handling characteristics during surgical procedures. The sensor module may be configured to track instrument movement, instrument position, or instrument usage patterns as a surgeon or trainee manipulates the instrumented surgical tool.
[0022] With continued reference to FIG. 1, wires may extend along a length of the instrumented surgical tool and may be connected to the electronic circuit board. The wires may enable signal transmission from the sensor module to an external recording system or processing system. In some embodiments, a twisted pair configuration may reduce electromagnetic interference during signal transmission, thereby improving data quality for subsequent analysis.
[0023] The instrumented surgical tool may function as a data collection apparatus that operates together with video-based surgical skill assessment systems. Data captured by the sensor module regarding instrument handling characteristics may be correlated with visual observations extracted from surgical video recordings. Such correlation may enable a more comprehensiveanalysis of surgical technique by combining kinematic data from the instrumented surgical tool with spatial and temporal features extracted from video frames.
[0024] In some embodiments of the invention, the instrumented surgical tool may capture data including grip force, rotation angle, translation velocity, or acceleration patterns during suturing tasks. The captured instrument data may be synchronized with video recordings to provide ground truth information for training machine learning models. The combination of instrument-based sensing and video-based analysis may enhance the accuracy of automated surgical skill assessment by providing complementary data streams that characterize different aspects of surgical performance.
[0025] Referring to FIGS. 2A and 2B, a video frame analysis display 200 depicts data capture within a surgical video frame that can be used by a machine learning system for automated skill assessment. FIG. 2A shows an image of hands manipulating surgical instruments over a surgical site with sterile draping visible. FIG. 2B shows corresponding labels of a video frame as a counterpart to the image of FIG. 2A. The video frame analysis display 200 presents visual output from the spatial feature extraction operations performed on recorded surgical procedures, enabling visualization of detected elements within the surgical field.
[0026] An action classification label 202 appears in an upper region of the video frame analysis display 200. The action classification label 202 displays a surgical activity, such as "suturing."
[0027] With continued reference to FIG. 2B, a hand detection bounding box 204 identifies and bounds detected hands within the surgical field. Multiple instances of the hand detection bounding box 204 may track different hands visible during the procedure, with each hand detection bounding box 204 surrounding a respective hand region. The hand detection bounding box 204 enables the system to isolate hand regions for subsequent keypoint analysis and movement tracking throughout the surgical video sequence.
[0028] An instrument detection bounding box 206 identifies surgical tools within the frame. The instrument detection bounding box 206 may use a dashed boundary or other visual indicator to distinguish instrument detection from hand detection. The instrument detection bounding box 206 may identify surgical instruments such as needle drivers, forceps, or other tools manipulated during the procedure. As further shown in FIG. 2B, the instrument detection bounding box 206provides spatial localization of surgical instruments relative to the detected hands, enabling analysis of hand-instrument coordination patterns.
[0029] Keypoint markers 210 represent tracked anatomical landmarks on detected hands and can map the hand pose and finger positions within the video frame analysis display 200. The keypoint markers 210 may appear as circular markers in various colors including red, blue, orange, and yellow to distinguish different anatomical points. The keypoint markers 210 enable analysis of hand positioning and movement patterns during surgical procedures by tracking specific locations on each detected hand across sequential frames.Categorizing Surgical Skill
[0030] Referring to FIG. 3, a data distribution visualization depicts the categorization of surgical skill assessment scores according to a Global Rating Score (GRS) scale. The GRS scale may range from 8 to 40 points and may be based on Objective Structured Assessment of Technical Skills (OSATS) evaluation performed by three independent raters. The horizontal axis of the visualization represents GRS values across the scoring range, while individual data points appear as markers distributed across the score range with a density curve overlaid to illustrate the distribution pattern.
[0031] The surgical skill assessment system may categorize performance into three skill levels based on mathematical thresholds applied to the GRS. A novice skill category may include performances where the GRS is less than 16 points. An intermediate skill category may encompass performances where the GRS is greater than or equal to 16 points and less than 24 points. An expert skill category may include performances where the GRS is greater than or equal to 24 points. In some cases, vertical demarcation lines at score values of 16 and 24 may visually separate the three skill categories within the distribution visualization.
[0032] With continued reference to FIG. 3, the dataset may a set of surgical training videos distributed across the three skill classification categories. The novice skill category may contain approximately 36.9 percent of the samples in the dataset. The intermediate skill category may contain approximately 25.2 percent of the samples. The expert skill category may contain approximately 37.9 percent of the samples. The distribution may exhibit a bimodal characteristic with concentrations of data points at both lower and higher score ranges, corresponding to the novice and expert skill categories respectively. The intermediate skill category may contain asmaller proportion of samples relative to the novice and expert categories. In some cases, the action classification label generated by the machine learning system may categorize surgical skill into one of the three levels based on the GRS thresholds derived from OSATS evaluation. While specific scores, bounds, and distributions are discussed above, one skilled in the art will recognize that these are examples and embodiments of the invention may contemplate other numerical values.Machine Learning Architecture
[0033] Referring to FIG. 4, a machine learning pipeline 400 for automated surgical skill assessment receives and processes surgical video recordings through a series of interconnected processing stages. An input frame sequence 402 receives video frames representing red, green, and blue color channels captured from surgical procedure recordings. Input frame sequence 402 may receive frames extracted from standard surgical video recordings captured using overhead camera configurations without requiring specialized tracking devices or sensor-instrumented tools. In some embodiment of the invention, the frames may have dimensions of 512x512x3 pixels.
[0034] In some embodiments, videos may be standardized to ensure uniform temporal representation for processing by the machine learning pipeline. Each video may be standardized to a particular duration (e.g., 5 minutes) and uniformly sampled at a particular rate (e.g., 1 frame per second). An example at a duration of 5 minutes and sampled 1 frame per second would correspond to extracting every 30th frame from videos recorded at 30 frames per second. This sampling strategy yields a fixed sequence of 300 frames per video without overlapping windows. Each frame may undergo standardization including resizing to the target input dimensions and normalization to ensure consistent dimensions across all inputs for the spatial feature extraction module.
[0035] A spatial feature extraction module 404 processes the video frames received from input frame sequence 402 to extract spatial features representing visual characteristics of surgical techniques. Spatial feature extraction module 404 may utilize a convolutional neural network architecture that has been adapted for surgical video analysis through domain-specific fine-tuning. In some embodiments, spatial feature extraction module 404 may employ a fine-tuned ResNet50 backbone with a custom classification head, though other convolutional neural network architectures such as VGG, EfficientNet, DenseNet, or vision transformers may alternativelyperform the spatial feature extraction function. The domain-specific fine-tuning enables spatial feature extraction module 404 to recognize subtle nuances in surgical techniques that differ from general object recognition tasks. Spatial feature extraction module 404 may include one or more convolution layers with activation functions and one or more pooling layers that process input frames to produce feature vectors for subsequent temporal analysis.
[0036] With continued reference to FIG. 4, spatial feature extraction module 404 may include a custom classification head incorporating Global Average Pooling and dense layers with L1-L2 regularization. The custom classification head enables spatial feature extraction module 404 to learn surgery-specific feature representations while preventing overfitting, which may be a consideration given limited availability of surgical training data. In some cases, the L1-L2 regularization applied to the dense layers constrains model parameters to improve generalization performance across different surgical video recordings.
[0037] During training, data augmentation techniques may be applied to introduce variability and improve model generalization. In some embodiments, augmentations may include horizontal flipping, brightness and / or contrast adjustments, rotation within a range such as ±15 degrees, and / or affine transformations including random shifts, scaling, and / or rotations. These transformations may help prevent overfitting and improve robustness of the spatial feature extraction module when processing video frames captured under varying conditions. The augmentation operations may be applied to training data while preserving the task-relevant visual features necessary for accurate skill classification.
[0038] While the disclosed embodiment utilizes a fine-tuned ResNet50 backbone, other convolutional neural network architectures may perform spatial feature extraction from video frames. In some cases, alternative architectures such as VGG, EfficientNet, DenseNet, or vision transformers may be employed within spatial feature extraction module 404. As shown in FIG. 4, the spatial features extracted by spatial feature extraction module 404 are passed to subsequent processing stages for temporal analysis.
[0039] A temporal feature extraction module 406 receives the spatial features from spatial feature extraction module 404 and analyzes sequential relationships between frames. Temporal feature extraction module 406 may employ a bidirectional LSTM (Long Short-Term Memory) architecture that processes surgical video sequences to understand both precedent and subsequent context of each surgical movement. The bidirectional nature of the LSTM within temporal featureextraction module 406 enables the model to analyze frame sequences in both forward and reverse temporal directions, creating a comprehensive understanding of surgical technique progression throughout a procedure.
[0040] The selection of LSTM architecture for temporal feature extraction module 406 may be motivated by several considerations. Surgical skill assessment may utilize analysis of extended temporal sequences where actions separated by multiple frames influence overall proficiency evaluation. The ability of LSTMs to maintain selective memory over long sequences may enable discrimination between efficient versus inefficient surgical workflows, where the temporal spacing and sequencing of movements distinguishes expert from novice performance. The bidirectional architecture may ensure comprehensive temporal context by processing sequences in both forward and backward directions, capturing dependencies that manifest across the entire surgical task rather than isolated segments.
[0041] Temporal feature extraction module 406 may incorporate a single-head soft attention mechanism that dynamically assigns importance weights to different segments of the surgical procedure. In some cases, the attention mechanism identifies crucial technical moments such as suture placements or tissue handling while maintaining awareness of overall procedural flow. The dynamic assignment of importance weights by the attention mechanism mirrors cognitive processes by which expert evaluators assess surgical procedures, where not all moments of a surgery are weighted equally during evaluation.
[0042] As further shown in FIG. 4, fully connected layers 408 receive processed temporal features from temporal feature extraction module 406 and generate classification outputs. A softmax activation 410 processes outputs from fully connected layers 408 to generate a probability distribution across skill categories. A probability distribution output 412 displays classification results for Novice, Intermediate, and Expert skill levels with corresponding probability values indicating confidence of classification for each skill category.Classification Output and Probability Distribution
[0043] Following temporal feature extraction, fully connected layers 408 may process the temporal features through a neural network structure. Fully connected layers 408 may receive the output from the temporal feature extraction module and transform the learned representations into a format suitable for classification. In some cases, fully connected layers 408 include one or moredense layers that progressively reduce the dimensionality of the feature space while preserving discriminative information relevant to surgical skill assessment. Fully connected layers 408 may incorporate regularization techniques such as L1-L2 regularization to prevent overfitting, particularly given the limited availability of surgical training data.
[0044] A softmax activation 410 may receive the output from fully connected layers 408 and generate a probability distribution across skill categories. Softmax activation 410 normalizes the output values such that the resulting probabilities sum to one, enabling interpretation of the model output as confidence scores for each skill classification. In some cases, softmax activation 410 applies an exponential function to each output value and divides by the sum of all exponential values to produce the normalized probability distribution.
[0045] A probability distribution output 412 may display classification results for Novice, Intermediate, and Expert skill levels with corresponding probability values. Probability distribution output 412 presents the confidence scores generated by softmax activation 410 in a format that indicates the likelihood of the assessed surgical performance belonging to each skill category. In some cases, probability distribution output 412 enables identification of the predicted skill level by selecting the category with the highest probability value. Probability distribution output 412 may also provide insight into classification uncertainty, where probability values distributed more evenly across categories indicate lower confidence
[0046] in the assessment compared to cases where one category receives a substantially higher probability value.
[0047] The temporal feature extraction phase may employ various sequence modeling architectures to analyze sequential relationships between frames and understand temporal patterns in surgical procedures. While bidirectional LSTM networks with attention mechanisms represent one implementation, other sequence modeling architectures may perform the temporal analysis function. In some cases, gated recurrent units (GRUs) may be employed as an alternative recurrent architecture for sequence analysis. Transformer architectures may be used in some embodiments to capture long-range dependencies between frames through self-attention mechanisms. Temporal convolutional networks may also be employed to extract temporal features through onedimensional convolution operations applied across the frame sequence. The selection of a particular sequence modeling architecture may depend on factors such as computational resources, dataset characteristics, and performance requirements for a given implementation.Training Strategy and Optimization
[0048] The machine learning pipeline may implement a two-phase training strategy to facilitate effective learning of surgical skill features from video data. During a first phase of training, a backbone network of the spatial feature extraction module may be frozen, meaning that weights of the pre-trained backbone network remain fixed and are not updated during backpropagation. Freezing the backbone network during the first phase allows other components of the machine learning pipeline, such as custom classification layers and temporal processing components, to learn appropriate feature representations without disrupting the pre-trained spatial features. The frozen backbone network provides stable feature extraction while downstream layers adapt to the surgical skill assessment task.
[0049] During a second phase of training, the backbone network may be unfrozen, allowing weights of the backbone network to be updated through backpropagation along with the remaining components of the machine learning pipeline. Unfreezing the backbone network during the second phase enables fine-tuning of the spatial feature extraction to learn domain-specific representations tailored to surgical video analysis. The two-phase training strategy may prevent catastrophic forgetting of pre-trained features while enabling the machine learning pipeline to adapt to the specific characteristics of surgical skill assessment data.
[0050] The machine learning pipeline may utilize 5-fold cross-validation during training to ensure consistent performance across different partitions of training data. In 5-fold cross-validation, a dataset may be divided into five approximately equal subsets or folds. For each iteration of cross-validation, four folds may be used for training while a remaining fold may be held out for validation. The training and validation process may be repeated five times, with each fold serving as the validation set exactly once. Performance metrics may be computed for each fold and aggregated to provide an overall assessment of model performance. The 5-fold cross-validation methodology may reduce variance in performance estimates and provide a more robust evaluation of the machine learning pipeline across different data partitions.
[0051] The machine learning pipeline may employ Bayesian hyperparameter optimization to tune model parameters. Bayesian hyperparameter optimization may utilize a probabilistic model to guide the search for hyperparameter configurations that yield improved performance. The probabilistic model may be updated based on results from previous hyperparameter evaluations,allowing subsequent evaluations to focus on regions of the hyperparameter space that are more likely to produce favorable results. Hyperparameters subject to optimization may include learning rates, regularization coefficients, layer dimensions, dropout rates, and other configurable parameters of the machine learning pipeline. Bayesian hyperparameter optimization may provide a more efficient search strategy compared to grid search or random search approaches by leveraging information from prior evaluations to inform subsequent hyperparameter selections.Model Performance Validation
[0052] Referring to FIG. 5, a Receiver Operating Characteristic (ROC) curve analysis demonstrates classification performance of the machine learning model across five-fold cross-validation in a certain embodiment of the invention. The graph plots True Positive Rate on a vertical axis against False Positive Rate on a horizontal axis, with both axes ranging from 0 to 1.0. Multiple curves represent ROC performance for each of five validation folds. ROC Fold 1 may exhibit an Area Under the Curve (AUC) of 0.96. ROC Fold 2 may exhibit an AUC of 0.94. ROC Fold 3 may exhibit an AUC of 0.96. ROC Fold 4 may exhibit an AUC of 0.96. ROC Fold 5 may exhibit an AUC of 0.95. Individual fold curves may be displayed with shaded regions indicating confidence intervals or variance around each curve.
[0053] A mean ROC curve may represent overall model performance averaged across all folds. The mean ROC AUC may be 0.95 ± 0.009, demonstrating consistent discriminative ability of the spatiotemporal deep learning architecture for automated surgical skill assessment. A diagonal dashed line extending from an origin to an upper right corner may represent performance of a random classifier, serving as a baseline reference. The individual fold curves may be positioned above the diagonal reference line and may be clustered together, indicating consistent classification performance across different data partitions.
[0054] The probability distribution output may achieve a macro-averaged precision of 0.82 for surgical skill classification. Skill-level classification performance may be characterized by Fl scores for each category. A novice skill level may achieve an Fl score of 0.91. An intermediate skill level may achieve an Fl score of 0.63. A proficient skill level may achieve an Fl score of 0.92. In some cases, misclassifications may occur between adjacent skill levels, while discrimination between novice and proficient levels may remain robust. The lower Fl score for intermediate classification may reflect inherent challenges in distinguishing subtle differencesbetween adjacent skill categories, mirroring evaluation patterns observed in expert human assessors.Attention Weight Distribution
[0055] The attention mechanism generates attention weight distributions that reveal distinct patterns between different skill levels. These attention weight distributions provide interpretable results that enable analysis of how the model evaluates surgical technique across temporal sequences.
[0056] Novice attention patterns exhibit characteristics associated with less developed surgical skills. Novice performers demonstrate extended preparation time during initial phases of surgical procedures. The attention weights for novice performers concentrate on segments involving multiple tool adjustments as the performer repositions instruments repeatedly before executing surgical maneuvers. Novice attention patterns further highlight repeated position checks where the performer pauses to verify hand placement or instrument orientation. Hesitation points appear as concentrated attention weights at moments where the performer stops or slows movement before proceeding with subsequent actions. The attention distribution for novice performers reflects an irregular rhythm characterized by variable speed and inconsistent timing throughout the procedure. Recovery time segments receive elevated attention weights as novice performers require additional time to correct positioning errors or adjust technique following suboptimal movements.
[0057] Expert attention patterns contrast with novice patterns through characteristics associated with developed surgical proficiency. Expert performers demonstrate efficient preparation with minimal setup time reflected in lower attention weights during initial procedural phases. The attention mechanism assigns weights to expert performances that reflect streamlined execution with fluid movements throughout the surgical sequence. Expert attention patterns indicate no recovery time segments as expert performers maintain optimal positioning without requiring corrective adjustments. The attention distribution for expert performers reflects a consistent rhythm with steady pace and predictable timing across the temporal sequence. Expert performances exhibit attention weight distributions that spread more uniformly across procedural segments rather than concentrating on preparation or recovery phases.
[0058] The attention weight distributions enable differentiation between adjacent skill levels by identifying temporal segments where technique differences manifest. Intermediate performers exhibit attention patterns that share characteristics with both novice and expert distributions, with some hesitation points and tool adjustments but with more consistent rhythm than novice performers. The interpretable nature of the attention mechanism allows identification of specific procedural moments that contribute to skill level classification, providing targeted feedback regarding portions of procedures that may benefit from additional training focus.System Integration
[0059] The machine learning pipeline implements efficient feature caching to create a practical, deployable system for automated surgical skill assessment. Feature caching stores extracted spatial features from the ResNet50 backbone after initial processing, eliminating redundant computation when analyzing video frames that have been previously processed. The cached features may be stored in local memory, distributed storage systems, or database architectures depending on deployment requirements. In some cases, the feature caching mechanism reduces computational overhead during inference by retrieving pre-computed feature vectors rather than re-processing raw video frames through the spatial feature extraction layers. The caching implementation may utilize hash-based indexing to enable rapid retrieval of stored features corresponding to specific video segments or individual frames.
[0060] The system integrates into existing surgical training programs with minimal infrastructure changes beyond standard video recording capabilities. In some cases, the system accepts video input from commercially available cameras positioned to capture bird's-eye view perspectives of surgical procedures. The video recording setup may utilize frame rates of approximately 30 frames per second, which provides sufficient temporal resolution for capturing surgical movements without requiring specialized high-speed imaging equipment. Integration with existing training workflows may involve connecting the system to institutional video storage systems or learning management platforms through standard application programming interfaces. The system may process videos in batch mode for retrospective assessment or may be configured for near-real-time analysis depending on available computational resources.
[0061] The system is designed to provide real-time assessment capabilities for surgical skill evaluation as a future enhancement. Real-time processing may involve streaming video framesdirectly to the spatial feature extraction module as frames are captured during live surgical procedures. In some cases, the temporal feature extraction module may operate on sliding windows of recent frames to generate continuous skill assessments throughout a procedure. The real-time assessment architecture may incorporate edge computing devices positioned within surgical training environments to reduce latency associated with transmitting video data to remote processing servers. Feedback mechanisms may display assessment results to trainees or instructors during or immediately following surgical exercises, enabling timely intervention and guidance based on detected skill deficiencies.System Operation
[0062] An automated surgical skill assessment system in many embodiments of the invention can operate through a coordinated sequence of processing stages that transform raw surgical video into objective skill classifications. Video capture devices positioned to obtain a bird's-eye view perspective record surgical procedures at frame rates suitable for capturing fine motor movements, such as 30 frames per second. The captured video undergoes preprocessing to extract individual frames at standardized dimensions, creating a sequential input suitable for machine learning analysis.
[0063] The input frame sequence receives video frames representing surgical procedure recordings and passes these frames to a spatial feature extraction module. The spatial feature extraction module employs a convolutional neural network architecture that has undergone domain-specific fine-tuning to recognize subtle nuances in surgical techniques. Rather than relying on generic image features, the spatial feature extraction module learns surgery-specific feature representations through transfer learning combined with custom classification layers. Global average pooling and dense layers with regularization enable the spatial feature extraction module to extract robust features while preventing overfitting given limited surgical training data availability. The spatial feature extraction module processes each frame independently to generate feature vectors encoding spatial characteristics of hand positions, instrument orientations, and tissue interactions visible within each frame.
[0064] The feature vectors from the spatial feature extraction module flow into a temporal feature extraction module that analyzes sequential relationships between frames. The temporal feature extraction module employs a bidirectional long short-term memory architecture thatprocesses the sequence of spatial features in both forward and reverse temporal directions. This bidirectional processing enables the temporal feature extraction module to understand both precedent and subsequent context of each surgical movement, creating a comprehensive representation of surgical technique that accounts for the progressive nature of procedural execution.
[0065] An attention mechanism integrated within the temporal feature extraction module dynamically assigns importance weights to different segments of the procedure. The attention mechanism mirrors the cognitive evaluation process of expert surgeons who do not weigh all moments of a surgery equally. During processing, the attention mechanism identifies technical moments such as suture placements, tissue handling maneuvers, and instrument transitions while maintaining awareness of overall procedural flow. The attention weights enable the system to focus on portions of the procedure that carry diagnostic value for skill assessment while contextualizing these moments within the broader surgical sequence.
[0066] The temporal features processed through the attention mechanism pass through fully connected layers that transform the temporal representations into classification-ready features. A softmax activation function generates a probability distribution across skill categories, producing a probability distribution output that displays classification results for novice, intermediate, and expert skill levels with corresponding probability values. The probability values indicate the system's confidence in each skill classification, enabling nuanced assessment that captures gradients of skill development rather than binary pass-fail determinations.
[0067] The interaction between the spatial and temporal processing stages enables the system to capture both the static quality of individual surgical moments and the dynamic quality of procedural execution overtime. The spatial feature extraction module identifies what is happening in each frame while the temporal feature extraction module with attention determines how the sequence of actions reflects underlying surgical competency. This hierarchical processing approach enables consistent, objective assessment that maintains logical coherence in classifications, with misclassifications occurring between adjacent skill levels rather than confusing novice and expert performances.
[0068] The disclosure of "Integration of spatiotemporal features into machine learning assessment of open surgical skills" to Alipour et al. is hereby incorporated by reference in its entirety. (Surgery. 2026 Jan 28; 192:110079).
[0069] Although the description above contains many specificities, these should not be construed as limiting the scope of the invention but as merely providing illustrations of some of the presently preferred embodiments of the invention. Various other embodiments are possible within its scope. Accordingly, the scope of the invention should be determined not by the embodiments illustrated, but by the appended claims and their equivalents.
Claims
WHAT IS CLAIMED IS:
1. A system for automated surgical skill assessment, comprising:a spatial feature extraction module configured to receive video frames captured from a surgical procedure and extract spatial features from the video frames using a convolutional neural network architecture that has undergone domain-specific fine-tuning for surgical video analysis;a temporal feature extraction module configured to receive the spatial features from the spatial feature extraction module and analyze sequential relationships between the video frames using a recurrent neural network architecture;an attention mechanism integrated within the temporal feature extraction module and configured to dynamically assign importance weights to different segments of the surgical procedure; anda classification module configured to receive temporal features from the temporal feature extraction module and generate a probability distribution across a plurality of skill categories indicating a skill level classification for the surgical procedure.
2. The system of claim 1, wherein the convolutional neural network architecture comprises a ResNet50 backbone with a custom classification head incorporating Global Average Pooling and dense layers with L1-L2 regularization.
3. The system of claim 2, wherein the domain-specific fine-tuning comprises a two-phase training strategy in which the ResNet50 backbone is frozen during a first phase and unfrozen during a second phase.
4. The system of claim 1 , wherein the recurrent neural network architecture comprises a bidirectional Long Short-Term Memory network configured to process the spatial features in both forward and reverse temporal directions.
5. The system of claim 4, wherein the attention mechanism comprises a single-head soft attention mechanism configured to identify technical moments including suture placements and tissue handling while maintaining awareness of overall procedural flow.
6. The system of claims 1-5, wherein the plurality of skill categories comprises a novice skill category, an intermediate skill category, and an expert skill category.
7. The system of claim 6, wherein the skill categories are defined based on a Global Rating Score derived from Objective Structured Assessment of Technical Skills evaluation, wherein the novice skill category corresponds to scores less than 16 points, the intermediate skill category corresponds to scores greater than or equal to 16 points and less than 24 points, and the expert skill category corresponds to scores greater than or equal to 24 points.
8. The system of claim 1, further comprising a feature caching module configured to store extracted spatial features from the spatial feature extraction module to eliminate redundant computation when analyzing previously processed video frames.
9. The system of claim 1, wherein the classification module comprises:fully connected layers configured to receive the temporal features and transform learned representations into a format suitable for classification; anda softmax activation function configured to generate the probability distribution across the plurality of skill categories.
10. The system of claim 1, wherein the video frames are captured from a bird's-eye view perspective at a frame rate of approximately 30 frames per second.
11. The system of claim 1, wherein the attention mechanism generates attention weight distributions that differentiate between skill levels by identifying temporal segments where technique differences manifest, wherein novice attention patterns exhibit extended preparation time and hesitation points, and wherein expert attention patterns exhibit efficient preparation and consistent rhythm.
12. A method for automated assessment of surgical skills, comprising:receiving a sequence of video frames captured from a surgical procedure;extracting spatial features from the video frames using a convolutional neural network that has been fine-tuned for surgical video analysis;processing the spatial features through a temporal feature extraction module comprising a recurrent neural network to analyze sequential relationships between the video frames;applying an attention mechanism to dynamically assign importance weights to different segments of the surgical procedure; andgenerating a probability distribution across a plurality of skill categories based on the temporal features to classify a skill level of the surgical procedure.
13. The method of claim 12, wherein the convolutional neural network comprises a ResNet50 backbone with a custom classification head incorporating Global Average Pooling and dense layers with L1-L2 regularization.
14. The method of claim 13, wherein fine-tuning the convolutional neural network comprises a two-phase training strategy in which the ResNet50 backbone is frozen during a first phase and unfrozen during a second phase.
15. The method of claim 12, wherein the recurrent neural network comprises a bidirectional Long Short-Term Memory network configured to process the spatial features in both forward and reverse temporal directions.
16. The method of claim 15, wherein applying the attention mechanism comprises applying a single-head soft attention mechanism configured to identify technical moments including suture placements and tissue handling while maintaining awareness of overall procedural flow.
17. The method of claims 12-16, wherein the plurality of skill categories comprises a novice skill category, an intermediate skill category, and an expert skill category defined based on a Global Rating Score derived from Objective Structured Assessment of Technical Skills evaluation.
18. The method of claim 12, further comprising caching extracted spatial features to eliminate redundant computation when analyzing previously processed video frames.
19. The method of claim 12, wherein applying the attention mechanism generates attention weight distributions that differentiate between skill levels by identifying temporal segments where technique differences manifest, wherein novice attention patterns exhibit extended preparation time, multiple tool adjustments, and hesitation points, and wherein expert attention patterns exhibit efficient preparation, streamlined execution, and consistent rhythm.
20. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising:receiving video frames captured from a surgical procedure;extracting spatial features from the video frames using a convolutional neural network architecture fine-tuned for surgical technique recognition;analyzing sequential relationships between the video frames by processing the spatial features through a bidirectional recurrent neural network;applying an attention mechanism to assign importance weights to segments of the surgical procedure; andgenerating a skill level classification for the surgical procedure based on a probability distribution across skill categories.
21. The non-transitory computer-readable medium of claim 20, wherein the convolutional neural network architecture comprises a ResNet50 backbone with a custom classification head incorporating Global Average Pooling and dense layers with L1-L2 regularization.
22. The non-transitory computer-readable medium of claim 21, wherein the operations further comprise training the convolutional neural network architecture using a two-phase training strategy in which the ResNet50 backbone is frozen during a first phase and unfrozen during a second phase.
23. The non-transitory computer-readable medium of claims 20-22, wherein the skill categories comprise a novice skill category corresponding to a Global Rating Score less than 16 points, an intermediate skill category corresponding to a Global Rating Score greater than or equal to 16 points and less than 24 points, and an expert skill category corresponding to a Global Rating Score greater than or equal to 24 points.
24. The non-transitory computer-readable medium of claim 20, wherein the attention mechanism comprises a single-head soft attention mechanism that generates attention weight distributions identifying temporal segments where technique differences manifest between skill levels.
25. The non-transitory computer-readable medium of claim 24, wherein the operations further comprise caching extracted spatial features to eliminate redundant computation when analyzing previously processed video frames.