Surgical video scene analysis method and apparatus, device, and program product

By constructing a large-scale surgical video dataset and performing spatiotemporal joint modeling, the problem of existing surgical scene understanding methods being unable to capture dynamic information was solved. This enabled collaborative learning of spatial and temporal information within a unified framework, improving the model's generalization ability and adaptability.

WO2026237742A1PCT designated stage Publication Date: 2026-11-19THE HONG KONG UNIV OF SCI & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2026/093962
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-16
Filing Date
2026-04-29
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

In existing technologies, AI-driven surgical scene understanding methods mainly perform spatial feature learning on static images, which makes it difficult to fully capture interactive information in dynamic surgical scenes, thus limiting the completeness and generalization ability of surgical scene understanding.

Method used

A large-scale surgical video dataset was constructed, and a video-level spatiotemporal representation was pre-trained using a video base model based on spatiotemporal joint modeling. The spatial structure information and temporal dynamic information were modeled simultaneously using an adaptive spatiotemporal masking mechanism and a multi-granularity distillation mechanism to obtain a pre-trained video understanding model. Subsequently, a video understanding task model was obtained through task transfer processing.

Benefits of technology

Under a unified framework, the collaborative learning of spatial and temporal information is realized, which improves the model's spatiotemporal abstraction ability and generalization performance under cross-task, cross-operative, and cross-scenario conditions, and can better adapt to various downstream tasks of surgical videos.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2026093962_19112026_PF_FP_ABST
    Figure CN2026093962_19112026_PF_FP_ABST
Patent Text Reader

Abstract

The present application discloses a surgical video scene analysis method and apparatus, a device, and a program product. The method comprises: constructing a large-scale surgical video dataset; inputting the large-scale surgical video dataset into a joint spatio-temporal modeling-based video foundation model for video-level spatio-temporal representation pre-training, and simultaneously modeling spatial structure information and temporal dynamic information by means of an adaptive spatio-temporal masking mechanism and a multi-granularity distillation mechanism to obtain a video understanding pre-trained model; acquiring surgical video understanding task data to perform task migration processing on the video understanding pre-trained model to obtain a video understanding task model; and inputting a surgical video to be analyzed into the video understanding task model for surgical scene analysis processing to obtain a surgical scene analysis result. Embodiments of the present application can improve the efficiency of surgical video scene analysis, and can be widely applied to the technical field of artificial intelligence.
Need to check novelty before this filing date? Find Prior Art

Description

A surgical video scene analysis method, device, equipment, and program product. Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a surgical video scene analysis method, device, equipment, and program product. Background Technology

[0002] In related technologies, AI-driven surgical scene understanding methods often employ self-supervised spatial representation learning to reduce the cost of manual annotation. However, these methods primarily perform spatial feature learning on static images during the pre-training stage, making it difficult for the model to fully capture interactive information in dynamic surgical scenes, thus limiting the completeness and generalization ability of surgical scene understanding.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0004] The main objective of this application is to provide a surgical video scene analysis method, apparatus, device, and program product that can improve the efficiency of understanding and analyzing surgical videos.

[0005] To achieve the above objectives, one aspect of this application proposes a surgical video scene analysis method, the method comprising:

[0006] Construct a large-scale surgical video dataset;

[0007] The large-scale surgical video dataset is input into the video base model based on spatiotemporal joint modeling for video-level spatiotemporal representation pre-training. Spatial structure information and temporal dynamic information are modeled simultaneously through an adaptive spatiotemporal masking mechanism and a multi-granularity distillation mechanism to obtain a video understanding pre-trained model.

[0008] The video understanding task data is obtained and the video understanding pre-trained model is subjected to task transfer processing to obtain the video understanding task model.

[0009] The surgical video to be analyzed is input into the video understanding task model for surgical scene analysis and processing to obtain the surgical scene analysis results.

[0010] In some embodiments, constructing a large-scale surgical video dataset includes the following steps:

[0011] Obtain publicly available surgical video data, and while retaining the original task division and original annotation information, resample and convert the encoding format of the publicly available surgical video data to obtain the first video dataset;

[0012] Acquire online surgical video data, identify effective segments in the online surgical video data used for surgical scene modeling, and obtain filtered video data;

[0013] The filtered video data is subjected to video frame visual content detection and audio risk processing to obtain privacy-protected video data;

[0014] Candidate surgical procedure keywords and anatomical structure keywords are extracted from the privacy-preserving video data, and standardized mapping and label construction are performed on the candidate surgical procedure keywords and anatomical structure keywords based on a pre-constructed vocabulary to obtain labeled video data;

[0015] Identify and filter duplicate or similar content in the tagged video data to obtain quality control video data;

[0016] The quality control video data is standardized to obtain a second video dataset;

[0017] The large-scale surgical video dataset is constructed based on the first video dataset and the second video dataset.

[0018] In some embodiments, the step of inputting the large-scale surgical video dataset into a video base model based on spatiotemporal joint modeling for video-level spatiotemporal representation pre-training, and simultaneously modeling spatial structure information and temporal dynamic information through an adaptive spatiotemporal masking mechanism and a multi-granularity distillation mechanism to obtain a video understanding pre-trained model includes the following steps:

[0019] Input video segments are selected from the large-scale surgical video dataset and input into the video base model, which includes a spatiotemporal joint modeling encoder, a video reconstruction decoder, and a feature alignment decoder.

[0020] The input video segment is divided into multiple non-overlapping spatiotemporal blocks arranged in an ordered manner along the time and spatial dimensions, and each spatiotemporal block is mapped to a corresponding visual word.

[0021] The spatiotemporal joint modeling encoder performs feature extraction and feature update processing on the visual lexical units. Based on the self-attention mechanism, it models the temporal association and spatial neighborhood relationship between the visual lexical units and fuses the temporal context information and spatial context information corresponding to each visual lexical unit to obtain a lexical feature representation containing spatiotemporal dependencies, thereby obtaining enhanced visible lexical units.

[0022] The input video segment is subjected to adaptive spatiotemporal masking to obtain masked terms;

[0023] The enhanced visible word and the mask word are concatenated to obtain the concatenated word;

[0024] The concatenated words are input into the video reconstruction decoder to recover the low-level pixel information of the occluded video area, and the reconstructed video data is output.

[0025] The concatenated lexical units are input into the feature alignment decoder for spatial feature alignment and multi-granularity distillation, and the output is an aligned feature representation.

[0026] The loss is calculated based on the reconstructed video data and the alignment feature representation, and the parameters of the video base model are updated based on the calculated loss value to obtain the video understanding pre-trained model.

[0027] In some embodiments, the adaptive spatiotemporal masking process for the input video segment to obtain masked terms includes the following steps:

[0028] The motion intensity, instrument activity density, tissue texture complexity, and degree of local scene change of the input video segment are analyzed to obtain the video content complexity;

[0029] The input video segment is masked by dynamically adjusting the mask position and mask ratio according to the complexity of the video content, thereby obtaining the masked words.

[0030] In some embodiments, the step of inputting the concatenated lexical units into the feature alignment decoder for spatial feature alignment and multi-granularity distillation, and outputting an aligned feature representation, includes the following steps:

[0031] The concatenated lexical units are input into the feature alignment decoder, and the output is the prediction space representation.

[0032] The spatial feature representation of each frame in the input video segment is obtained through an image-level teacher model, and the overall temporal feature representation of the input video segment is obtained through a video-level teacher model.

[0033] The predicted spatial representation output by the video base model is aligned with the spatial feature representation and the temporal feature representation, respectively. By mapping the predicted features output by the video base model and the features output by the teacher model to the same feature space, and calculating the difference between features based on the loss function, the feature distribution is aligned by minimizing the difference, and the aligned feature representation is obtained.

[0034] In some embodiments, the step of calculating loss based on the reconstructed video data and the alignment feature representation, and updating the parameters of the video base model based on the calculated loss value to obtain the video understanding pre-trained model includes the following steps:

[0035] The reconstruction loss is calculated based on the reconstructed video data.

[0036] Distillation loss is calculated by performing distillation loss on the aligned feature representation based on a preset feature alignment loss function to obtain the distillation loss.

[0037] By assigning corresponding domain identifiers to surgical video data from different sources in the large-scale surgical video dataset, and performing cross-domain feature matching on video segments with the same surgical procedure category or similar anatomical structures, the feature distribution difference between different domains is calculated based on the maximum mean difference loss or contrastive learning loss, and the domain alignment loss is obtained by minimizing the feature distribution difference.

[0038] The samples are evaluated based on the predicted loss value or prediction confidence during the training process of the model, and samples with loss values ​​higher than a preset threshold or prediction confidence lower than a preset threshold are identified as difficult samples. The sampling weight or supervision intensity is increased for the difficult samples to obtain the temporal consistency constraint.

[0039] The parameters of the video base model are updated based on the reconstruction loss, the distillation loss, the domain alignment loss, and the temporal consistency constraint to obtain the video understanding pre-trained model.

[0040] In some embodiments, the step of acquiring surgical video understanding task data and performing task transfer processing on the video understanding pre-trained model to obtain a video understanding task model includes the following steps:

[0041] A task header is constructed based on the surgical video understanding task data;

[0042] The surgical video understanding task data is input into the spatiotemporal joint modeling encoder of the video understanding pre-trained model, and the output is a video-level spatiotemporal feature representation.

[0043] Based on the task head, the video-level spatiotemporal feature representation is output according to the task objective. Based on the output results, the spatiotemporal joint modeling encoder and the task head are trained under supervision to obtain the video understanding task model.

[0044] To achieve the above objectives, another aspect of this application provides a surgical video scene analysis device, the device comprising:

[0045] The data building module is used to construct large-scale surgical video datasets;

[0046] The pre-training module is used to input the large-scale surgical video dataset into the video base model based on spatiotemporal joint modeling for video-level spatiotemporal representation pre-training. Through an adaptive spatiotemporal masking mechanism and a multi-granularity distillation mechanism, spatial structure information and temporal dynamic information are modeled simultaneously to obtain a video understanding pre-trained model.

[0047] The task transfer module is used to acquire surgical video understanding task data and perform task transfer processing on the video understanding pre-trained model to obtain a video understanding task model.

[0048] The scene analysis module is used to input the surgical video to be analyzed into the video understanding task model for surgical scene analysis processing, and to obtain the surgical scene analysis results.

[0049] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0050] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0051] The embodiments of this application include at least the following beneficial effects: This application provides a surgical video scene analysis method, apparatus, device, and program product. By constructing a large-scale surgical video dataset, this application can provide a high-quality data foundation for robust and generalizable spatiotemporal representation learning. Furthermore, by inputting the large-scale surgical video dataset into a video-based base model based on spatiotemporal joint modeling for video-level spatiotemporal representation pre-training, this application obtains a video understanding pre-trained model. This model can simultaneously model spatial structural information and temporal dynamic information within a unified pre-training framework, thereby enabling the learning of robust, transferable, and discriminative spatiotemporal feature representations in diverse surgical scenarios. This allows the model to complete the collaborative learning of spatial and temporal information within a unified framework. Additionally, this application performs task transfer processing on the video understanding pre-trained model by acquiring surgical video understanding task data, obtaining a video understanding task model. This allows the pre-trained model to more efficiently adapt to various downstream tasks of surgical videos and improves the spatiotemporal modeling capability and generalization performance of complex intraoperative scenarios. Attached Figure Description

[0052] Figure 1 is a schematic diagram of an implementation environment provided in an embodiment of this application;

[0053] Figure 2 is a flowchart of a surgical video scene analysis method provided in an embodiment of this application;

[0054] Figure 3 is a flowchart of step S201 in Figure 2;

[0055] Figure 4 is a flowchart of step S202 in Figure 2;

[0056] Figure 5 is a flowchart of step S404 in Figure 4;

[0057] Figure 6 is a flowchart of step S407 in Figure 4;

[0058] Figure 7 is a flowchart of step S408 in Figure 4;

[0059] Figure 8 is a flowchart of step S203 in Figure 2;

[0060] Figure 9 is a schematic diagram of the structure of a surgical video scene analysis device provided in an embodiment of this application;

[0061] Figure 10 is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0063] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0064] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0066] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0067] 1) Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence. It involves perceiving the environment, acquiring knowledge, and using that knowledge to obtain optimal results.

[0068] 2) Self-supervised learning is a technique that lies between supervised and unsupervised learning. Its core idea is to use the structure or attributes of the data itself to generate "pseudo-labels," transforming the unsupervised problem into a supervised problem, thereby training the model to learn effective feature representations.

[0069] Computer-aided intervention technology has been widely applied in modern surgical procedures. Among these, surgical scenario understanding is a crucial technological foundation for supporting intraoperative decision-making, improving surgical efficiency, and ensuring intraoperative safety. Surgical scenario understanding typically requires the automatic analysis of instrument status, anatomical structures, maneuvers, surgical stages, and skill performance in surgical videos. Therefore, it is highly dependent on both spatial and temporal information in the video data.

[0070] In related technologies, artificial intelligence methods are largely limited by the scale of labeled data. Due to the wide variety of surgical procedures and the significant differences in their prevalence, operational processes, technical complexity, and anatomical objects, surgical video data exhibits a significant long-tail distribution. Collecting sufficient labeled video data for such a broad and unevenly distributed range of surgical types is not only costly but also requires the participation of numerous professionals for annotation. Therefore, relevant public datasets typically only cover a limited number of surgical procedures and scenarios. Models trained in this way are often limited to specific tasks and surgical procedures, lacking generalization ability when facing different surgical types, equipment conditions, and scenario complexities, making it difficult to meet the practical needs of clinical applications across surgical procedures, scenarios, and tasks.

[0071] To reduce reliance on manual annotation, self-supervised learning has gradually become an important technological direction in the field of medical intelligent analysis. Related research has demonstrated in medical fields such as radiology and pathology that large-scale unlabeled data can be used to learn robust and transferable feature representations. However, related surgical self-supervised learning methods mainly focus on image-level pre-training, with their core objective remaining the learning of spatial representations of single-frame images, lacking a unified modeling approach for the crucial temporal dynamics in surgical videos. A common practice in related technical solutions is to first obtain static spatial features through image-level pre-training, and then introduce an additional temporal module in the downstream fine-tuning stage to perform temporal modeling on the pre-extracted spatial features. This two-stage modeling approach has the following shortcomings: First, the modeling objects in the pre-training and fine-tuning stages are inconsistent, leading to a disconnect between spatial representation learning and temporal dependency modeling, making it difficult to form a unified spatiotemporal representation. Second, the temporal module is usually trained from random initialization in the fine-tuning stage, failing to fully inherit the advantages of the large-scale data in the pre-training stage, resulting in temporal dependencies still mainly relying on limited downstream labeled data for learning. Third, in real surgical videos, there are often complex situations such as instrument occlusion, motion blur, smoke pollution, field of view switching, and differences in surgical procedures. If spatial and temporal information cannot be modeled collaboratively in the pre-training stage, it is difficult to achieve a stable understanding of the surgical scene. Therefore, the relevant technologies fail to fully utilize the rich temporal dynamic information contained in large-scale surgical video data, limiting the robustness and generalization ability of the model in complex surgical scenarios.

[0072] In view of this, this application provides a surgical video scene analysis method, apparatus, device, and program product. This application achieves a technological shift from static image modeling to dynamic video-level representation learning by directly executing explicit and unified spatiotemporal representation learning during the pre-training stage. This application constructs a large-scale publicly available surgical video dataset containing 3,652 videos and approximately 3.56 million frames, covering more than 20 surgical procedures and various anatomical structures, serving as the data foundation for general video-level modeling. Based on this, a video understanding pre-training model is proposed, directly encoding fine anatomical details and the temporal dynamics of the surgical process during the pre-training stage, enabling the model to complete the collaborative learning of spatial and temporal information within a unified framework. This application comprehensively validates the model on 12 datasets covering five surgical procedures and four types of video-level tasks. The results show that the proposed model exhibits stronger spatiotemporal abstraction and generalization capabilities under cross-task, cross-operative, and cross-scene conditions, effectively alleviating the long-standing data bottleneck and fragmented modeling problems in the field of surgical artificial intelligence.

[0073] This application provides a surgical video scene analysis method, which relates to the field of artificial intelligence technology. The surgical video scene analysis method provided in this application can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing a surgical video scene analysis method, but is not limited to the above forms.

[0074] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0075] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0076] Figure 1 is a schematic diagram of the implementation environment of a method provided in this application. Referring to Figure 1, the main hardware and software components of this implementation environment include a terminal 101 and a server 102, which are communicatively connected. The method can be executed based on the interaction between the terminal 101 and the server 102. Furthermore, the terminal 101 and the server 102 can be nodes in a blockchain; this embodiment does not specifically limit this.

[0077] Figure 2 is an optional flowchart of a surgical video scene analysis method provided in an embodiment of this application. The method in Figure 2 may include, but is not limited to, steps S201 to S204.

[0078] Step S201: Construct a large-scale surgical video dataset;

[0079] Step S202: Input the large-scale surgical video dataset into the video basic model based on spatiotemporal joint modeling for video-level spatiotemporal representation pre-training. Simultaneously model spatial structure information and temporal dynamic information through adaptive spatiotemporal masking mechanism and multi-granularity distillation mechanism to obtain video understanding pre-trained model.

[0080] Step S203: Obtain surgical video understanding task data and perform task transfer processing on the video understanding pre-trained model to obtain a video understanding task model;

[0081] Step S204: Input the surgical video to be analyzed into the video understanding task model for surgical scene analysis and processing to obtain the surgical scene analysis results.

[0082] Steps S201 to S204, as illustrated in this embodiment, firstly, a large-scale surgical video dataset is constructed. Then, a spatiotemporal joint encoding of video segments from the dataset is performed using a video foundation model based on spatiotemporal joint modeling. Training inputs for partially missing spatiotemporal regions are generated through mask modeling. Next, video reconstruction is performed on the occluded video regions based on a reconstruction decoder. Simultaneously, joint optimization is performed using feature distillation, cross-source alignment, and hard example reinforcement training. Finally, the pre-trained unified video-level representation is transferred to multiple surgical video understanding tasks to form a unified downstream application system. This application obtains robust, transferable, and discriminative video-level spatiotemporal feature representations by uniformly modeling the spatial structure and temporal dynamics of surgical scenes, thereby improving surgical video understanding capabilities and supporting tasks such as action recognition, surgical stage recognition, triplet recognition, and surgical skill evaluation.

[0083] One of the above technical solutions has the following advantages or beneficial effects: This application forms a general video-level basic model solution for complex surgical scenarios by constructing large-scale surgical video data, unified spatiotemporal pre-training and multi-task transfer application, providing unified spatiotemporal representation support for scene analysis tasks such as action recognition, surgical stage recognition, triplet recognition and surgical skill evaluation, and providing a scalable technical foundation for the development of subsequent intelligent surgical systems.

[0084] Referring to Figure 3, in step S201 of some embodiments, constructing a large-scale surgical video dataset includes the following steps:

[0085] Step S301: Obtain publicly available surgical video data. Based on retaining the original task division and original annotation information, resample and convert the encoding format of the publicly available surgical video data to obtain the first video dataset.

[0086] Step S302: Obtain online surgical video data, identify effective segments in the online surgical video data used for surgical scene modeling, and obtain filtered video data;

[0087] Step S303: Perform video frame visual content detection and audio risk processing on the screened video data to obtain privacy-protected video data;

[0088] Step S304: Extract candidate surgical procedure keywords and anatomical structure keywords from the privacy-preserving video data, and perform standardized mapping and label construction on the candidate surgical procedure keywords and anatomical structure keywords based on a pre-constructed vocabulary to obtain labeled video data;

[0089] Step S305: Identify and filter duplicate or similar content in the tagged video data to obtain quality control video data;

[0090] Step S306: Standardize the quality control video data to obtain a second video dataset;

[0091] Step S307: Construct the large-scale surgical video dataset based on the first video dataset and the second video dataset.

[0092] In this embodiment, a publicly available large-scale surgical video dataset is constructed, comprising a first video dataset and a second video dataset. The first video dataset consists of publicly available surgical video datasets, while the second video dataset consists of laparoscopic and endoscopic surgical videos obtained from publicly available online platforms. These two datasets together cover various anatomical structures and surgical procedures, providing data diversity and scene complexity for video-level spatiotemporal representation learning.

[0093] In step S301 of some embodiments, publicly available surgical video data is obtained, and the publicly available surgical video data is resampled and encoded in a different format while retaining the original task division and original annotation information to obtain a first video dataset.

[0094] For example, this application obtains publicly available surgical video data from publicly available surgical video datasets. While preserving the original task division and annotation information, it performs uniform standardization processing on all videos to obtain a first video dataset. The standardization processing in this application includes uniform resampling and uniform encoding format conversion on all videos.

[0095] Specifically, this application reduces temporal redundancy, balances the video length distribution across different datasets, and controls computational costs during the pre-training phase by uniformly resampling all videos to 1fps. Apart from resampling and unified encoding format, the dataset does not additionally perform key anatomical structure recognition or surgical type recognition; the relevant labels use the annotation information provided by the original publicly available dataset to preserve the original task definition and supervision signals.

[0096] In step S302 of some embodiments, network surgical video data is acquired, and effective segments in the network surgical video data used for surgical scene modeling are identified to obtain filtered video data.

[0097] For example, this application acquires online surgical video data by obtaining laparoscopic and endoscopic surgical videos from publicly available online platforms, covering more than twenty surgical procedures and involving multiple surgical specialties. This application establishes a video content understanding process to identify whether candidate videos in the online surgical video data contain valid segments that can be used for surgical scene modeling.

[0098] Specifically, this application employs shot type recognition technology to initially classify candidate videos, dividing the video content into intraoperative endoscopic footage, extraoperative camera footage, transition footage, and static title pages, retaining only continuous intraoperative endoscopic field-of-view segments. By identifying effective segments in online surgical video data used for surgical scene modeling and using these segments as screening video data, this application can exclude non-intraoperative content irrelevant to the target task in the early stages of data construction, thereby improving sample effectiveness.

[0099] In step S303 of some embodiments, video frame visual content detection and audio risk processing are performed on the screened video data to obtain privacy-protected video data;

[0100] For example, this application constructs a multi-stage de-identification processing flow to perform systematic screening and de-identification processing on candidate videos in the screened video data, wherein the de-identification processing includes visual content detection of video frames and audio risk processing.

[0101] Specifically, video frame visual content detection includes face detection, skin exposure area detection, overlaid text detection, hospital logo detection, and subtitle area detection, used to identify risky scenes that may reveal patient identities, medical staff identities, medical institution information, or other identifiable content. Audio risk processing involves removing or screening the original audio track to avoid retaining audio content containing names of people, institutions, or other sensitive information. For video segments that trigger any risk condition, a review mechanism combining automatic filtering rules and manual review is used to retain only continuous intraoperative segments that pass all privacy screening conditions for subsequent processing. After the above processing, the samples finally included in the first dataset are all fully de-identified intraoperative endoscopic images.

[0102] In step S304 of some embodiments, candidate surgical procedure keywords and anatomical structure keywords are extracted from the privacy-preserving video data, and the candidate surgical procedure keywords and anatomical structure keywords are standardized and mapped and labeled based on a pre-constructed vocabulary to obtain labeled video data;

[0103] For example, this application generates candidate tags based on text information associated with videos in privacy-preserving video data. Specifically, it extracts candidate surgical technique keywords and anatomical structure keywords from metadata such as video titles, video descriptions, and platform tags, and performs standardized mapping based on pre-built surgical technique vocabularies and anatomical structure vocabularies to unify the tag representation under different expression methods, thereby improving tag coverage and consistency.

[0104] In step S305 of some embodiments, duplicate or similar content in the tagged video data is identified and filtered to obtain quality control video data;

[0105] For example, this application reduces the interference caused by duplicate uploads, repeated segments, or excessive similar content in the network source data to training by performing deduplication and near-duplication filtering on candidate videos in the labeled video data. Specifically, this application uses techniques such as global hash discrimination, segment fingerprint matching, and keyframe visual embedding similarity retrieval to determine the degree of duplication between videos. When there are highly overlapping segments in videos from different sources, only samples with higher information completeness, higher image clarity, or better resolution are retained as the final data to reduce redundancy and improve the overall quality of the training data.

[0106] In step S306 of some embodiments, the quality control video data is standardized to obtain a second video dataset;

[0107] For example, this application performs uniform resampling and format normalization on videos in quality control video data that have passed screening and cleaning. Specifically, this application reduces temporal redundancy and maintains consistency across samples from different sources by uniformly resampling all videos to 1fps and converting them to a uniform encoding format. In a further implementation, a frame extraction strategy based on information density is adopted to perform differentiated sampling for different time segments. Specifically, a lower sampling frequency is used for low-dynamic stages, while the sampling rate is increased for high-dynamic stages such as instrument switching, bleeding, suturing, and clamping, and the sampling rate is determined by the optical flow intensity. Through the above processing, this application retains more key dynamic information while controlling computational costs, thereby improving the data's ability to support spatiotemporal representation learning tasks.

[0108] In step S307 of some embodiments, the large-scale surgical video dataset is constructed based on the first video dataset and the second video dataset.

[0109] For example, this application constructs a large-scale surgical video dataset by using video data from a first video dataset and a second video dataset. This large-scale surgical video dataset contains intraoperative surgical endoscopic images without identifiable patient information. The final constructed dataset covers more than 20 surgical procedures on more than 10 anatomical structures.

[0110] One of the above technical solutions has the following advantages or beneficial effects: This application improves the model's ability to model temporal dependencies by screening continuous intraoperative endoscopic segments to ensure the integrity of temporal dynamics; it enhances the model's spatial representation ability by covering multiple surgical procedures and anatomical structures with data; and it improves the model's ability to discriminate key surgical events and its overall generalization performance by reducing redundant information and highlighting key dynamic regions through deduplication, quality screening, and differential sampling strategies. This application, by constructing a large-scale surgical video dataset, can provide a high-quality data foundation for robust and generalizable spatiotemporal representation learning, and can be used to support the pre-training of basic surgical video models and the development and optimization of subsequent intelligent surgical analysis models.

[0111] Referring to Figure 4, in step S202 of some embodiments, the process of inputting the large-scale surgical video dataset into a video base model based on spatiotemporal joint modeling for video-level spatiotemporal representation pre-training, and simultaneously modeling spatial structure information and temporal dynamic information through an adaptive spatiotemporal masking mechanism and a multi-granularity distillation mechanism to obtain a video understanding pre-trained model includes the following steps:

[0112] Step S401: Select an input video segment from the large-scale surgical video dataset and input the input video segment into the video base model. The video base model includes a spatiotemporal joint modeling encoder, a video reconstruction decoder, and a feature alignment decoder.

[0113] Step S402: Divide the input video segment into multiple non-overlapping spatiotemporal blocks arranged in an orderly manner along the time and spatial dimensions, and map each spatiotemporal block to a corresponding visual word.

[0114] Step S403: The spatiotemporal joint modeling encoder performs feature extraction and feature update processing on the visual lexical units. Based on the self-attention mechanism, the temporal association and spatial neighborhood relationship between the visual lexical units are modeled, and the temporal context information and spatial context information corresponding to each visual lexical unit are fused to obtain a lexical feature representation containing spatiotemporal dependencies, thereby obtaining enhanced visible lexical units.

[0115] Step S404: Perform adaptive spatiotemporal masking on the input video segment to obtain masked terms;

[0116] Step S405: Concatenate the enhanced visible word and the mask word to obtain the concatenated word;

[0117] Step S406: Input the concatenated words into the video reconstruction decoder to recover the low-level pixel information of the occluded video area, and output the reconstructed video data.

[0118] Step S407: Input the concatenated lexical units into the feature alignment decoder for spatial feature alignment and multi-granularity distillation processing, and output the aligned feature representation;

[0119] Step S408: Based on the reconstructed video data and the alignment feature representation, perform loss calculation, and update the parameters of the video base model based on the calculated loss value to obtain the video understanding pre-trained model.

[0120] In this application, to break through the pre-training paradigm based on static images and establish video-level representation learning as the foundation of surgical artificial intelligence, this application proposes a basic model architecture for spatiotemporal joint modeling of surgical videos. Compared with methods that rely solely on static frame-level representations, the video understanding pre-training model of this application simultaneously models spatial structure information and temporal dynamic information within a unified pre-training framework, thereby enabling it to learn robust, transferable, and discriminative spatiotemporal feature representations in diverse surgical scenarios.

[0121] In step S401 of some embodiments, an input video segment is selected from a large-scale surgical video dataset and input into the video base model. The video understanding pre-trained model of this application includes a spatiotemporal joint modeling encoder for joint spatiotemporal dependency modeling, a video reconstruction decoder for recovering occluded video regions, and a feature alignment decoder for performing feature alignment and spatial representation refinement.

[0122] In step S402 of some embodiments, the input video segment given by this application is X. vv ∈R TTHHWW3 Using a cube of size P T× P H ×P WW The spatiotemporal joint cube embedding strategy divides video segments into multiple non-overlapping spatiotemporal blocks arranged in an ordered manner along the temporal and spatial dimensions, and maps each spatiotemporal block to a corresponding visual lexical unit, thereby obtaining a spatiotemporal representation. in, K represents the number of spatial lexical units at each time step, and C represents the dimension of lexical features.

[0123] In step S403 of some embodiments, the present application inputs visual lexical units into the spatiotemporal joint modeling encoder. The spatiotemporal joint modeling encoder can be constructed based on the ViT-B architecture and introduces a joint spatiotemporal attention mechanism in the self-attention calculation process to simultaneously capture inter-frame dynamic dependencies and intra-frame spatial organization structure, thereby obtaining enhanced visible lexical representations.

[0124] Referring to Figure 5, in step S404 of some embodiments, the adaptive spatiotemporal masking process performed on the input video segment to obtain masked terms includes the following steps:

[0125] Step S501: Analyze the motion intensity, instrument activity density, tissue texture complexity, and degree of local scene change of the input video segment to obtain the video content complexity;

[0126] Step S502: Dynamically adjust the mask position and mask ratio according to the complexity of the video content to perform masking processing on the input video segment, and obtain the mask words.

[0127] For example, this application constructs training input with partially missing spatiotemporal regions by performing mask modeling on the input video segments. This application can employ a uniform tubular masking strategy, i.e., maintaining consistent masking positions between adjacent frames and using a high masking ratio to mask the input video segments, thereby reducing information leakage and facilitating the model's learning of semantically meaningful spatiotemporal relationships from limited visible regions. Furthermore, this application can further employ an adaptive spatiotemporal masking strategy, dynamically adjusting the mask position and masking ratio according to the complexity of the video content.

[0128] Specifically, this application assigns different mask weights to different spatiotemporal regions based on the motion intensity, instrument activity density, tissue texture complexity, and degree of local scene change in video clips; it increases the mask ratio for low motion redundancy segments to enhance the model's ability to model long-range dependencies and global semantic structures; it reduces the mask ratio for high-dynamic key segments such as rapid instrument movement, bleeding, suturing, clamping, or tissue exposure to retain more key dynamic information; and it constrains the mask distribution based on the anatomical region localization results, using a lower occlusion ratio for key anatomical regions and a higher occlusion ratio for background regions or regions with low information density.

[0129] One of the above technical solutions has the following advantages or beneficial effects: This application, through adaptive spatiotemporal masking, enables the model to maintain the advantages of high-mask pre-training while taking into account both temporal dynamic modeling and preservation of key spatial details.

[0130] In step S405 of some embodiments, the present application concatenates the enhanced visible lexical output of the spatiotemporal joint modeling encoder with the learnable masked lexical obtained by adaptive spatiotemporal masking processing to obtain concatenated lexical.

[0131] In step S406 of some embodiments, this application inputs concatenated terms into a video reconstruction decoder to recover low-level pixel information of the occluded video region and outputs reconstructed video data. The video reconstruction decoder consists of multiple Transformer layers, used to infer pixel-level content of the missing region based on partially visible input, while maintaining consistency in spatial structure and temporal continuity of the reconstruction result. This application uses mask modeling to generate training input with partially missing spatiotemporal information, while video reconstruction is used to recover the content of the occluded region based on the missing input. Together, they constitute a self-supervised training mechanism for video-level spatiotemporal representation learning.

[0132] Referring to Figure 6, in step S407 of some embodiments, the concatenated lexical units are input into the feature alignment decoder for spatial feature alignment and multi-granularity distillation processing, and the output is an aligned feature representation, including the following steps:

[0133] Step S601: Input the concatenated lexical units into the feature alignment decoder and output the prediction space representation;

[0134] Step S602: Obtain the spatial feature representation of each frame in the input video segment through the image-level teacher model, and obtain the overall temporal feature representation of the input video segment through the video-level teacher model;

[0135] Step S603: Align the predicted spatial representation output by the video base model with the spatial feature representation and the temporal feature representation respectively. By mapping the predicted features output by the video base model and the features output by the teacher model to the same feature space, and calculating the difference between features based on the loss function, the feature distribution is aligned by minimizing the difference to obtain the aligned feature representation.

[0136] For example, this application introduces a multi-granularity distillation mechanism to compensate for the loss of fine-grained spatial details caused by cube embedding and temporal downsampling. This application can use a teacher model to distill a student model, thereby enhancing its ability to perceive fine-grained spatial information such as anatomical textures, tissue boundaries, and instrument contours. The teacher model includes both image-level and video-level teacher models, while the student model is a pre-built video-based model. This application introduces a video-level teacher model to provide supervision information on stage transitions, action continuity, and causal dynamic relationships. The student model simultaneously aligns frame-level spatial features, segment-level features, and video-level global features, thus forming a multi-granularity distillation framework. Different loss functions and loss weights are set for feature alignment at different granularities to balance the ability to reconstruct detailed textures with long-term dynamic modeling capabilities.

[0137] One of the above technical solutions has the following advantages or beneficial effects: Through the above-mentioned multi-teacher, multi-level distillation, the model simultaneously obtains fine-grained anatomical discrimination ability and high-level temporal understanding ability, thereby improving the integrity and transferability of the pre-trained representation.

[0138] Referring to Figure 7, in step S408 of some embodiments, loss calculation is performed based on the reconstructed video data and the alignment feature representation, and the parameters of the video base model are updated based on the calculated loss value to obtain the video understanding pre-trained model, including the following steps:

[0139] Step S701: Calculate the reconstruction loss function based on the reconstructed video data to obtain the reconstruction loss;

[0140] Step S702: Calculate the distillation loss on the aligned feature representation based on a preset feature alignment loss function to obtain the distillation loss;

[0141] Step S703: Assign corresponding domain identifiers to surgical video data from different sources in the large-scale surgical video dataset, perform cross-domain feature matching on video segments with the same surgical procedure category or similar anatomical structure, calculate the feature distribution difference between different domains based on the maximum mean difference loss or contrastive learning loss, and obtain the domain alignment loss by minimizing the feature distribution difference.

[0142] Step S704: Evaluate the samples based on the predicted loss value or prediction confidence of the model during the training process, and determine the samples with loss values ​​higher than a preset threshold or prediction confidence lower than a preset threshold as difficult samples, and increase the sampling weight or supervision intensity for the difficult samples to obtain temporal consistency constraints.

[0143] Step S705: Update the parameters of the video base model based on the reconstruction loss, the distillation loss, the domain alignment loss, and the temporal consistency constraint to obtain the video understanding pre-trained model;

[0144] For example, this application calculates reconstruction loss on reconstructed video data, where the reconstruction loss is defined as the mean square error between the normalized true pixel value and the reconstructed pixel value, and is calculated only within the masked region. By reconstructing missing regions in the entire video segment, this application guides the model to focus on temporal changes, instrument interaction patterns, and anatomical exposure processes in the surgical scene, thereby learning a spatiotemporal representation with global semantic constraints. This application calculates distillation loss based on feature alignment loss function to align feature representations, where the feature alignment loss function can be defined using Smooth L1 loss, L2 loss, and cosine similarity loss. Furthermore, considering the significant differences between different sources of training data in terms of acquisition devices, image style, noise type, annotation granularity, and scene distribution, this application further introduces a cross-source consistency training mechanism to improve the cross-domain generalization ability of the pre-trained representation. Specifically, this involves constructing domain identifiers for samples in a large-scale surgical video dataset, introducing domain alignment constraints during training to make segments from different sources but belonging to the same surgical procedure category, similar anatomical structures, or similar operational procedures closer together in the feature space, and achieving inter-domain representation alignment by minimizing cross-domain feature distance, maximum mean difference loss, or contrastive learning loss. Through cross-source consistency training, the model learns more stable surgical semantic representations and enhances its unified modeling ability for multi-source video data. Furthermore, this application introduces a hard example mining mechanism to improve the model's ability to discriminate between scenes with blurred boundaries and highly confusing scenes. Specifically, this includes continuously monitoring sample loss, prediction confidence, and temporal consistency indices. High-loss samples, boundary stage segments, rapid instrument entry and exit segments, bleeding obstruction segments, visual field contamination segments, and continuous segments with unstable predictions are identified as difficult samples. The sampling weight of the identified difficult samples is dynamically increased, and the supervision intensity of the corresponding segments is increased in the loss function to encourage the model to focus on learning the discriminative features of easily confused scenarios. Temporal consistency constraints are introduced for continuous segments whose prediction results fluctuate greatly within adjacent time windows. By limiting abnormal jumps in the features or output distribution of adjacent segments, the model's ability to stably understand continuous surgical procedures is enhanced.

[0145] Referring to Figure 8, in step S203 of some embodiments, the process of acquiring surgical video understanding task data and performing task transfer processing on the video understanding pre-trained model to obtain a video understanding task model includes the following steps:

[0146] Step S801: Construct a task head based on the surgical video understanding task data;

[0147] Step S802: Input the surgical video understanding task data into the spatiotemporal joint modeling encoder of the video understanding pre-trained model, and output the video-level spatiotemporal feature representation;

[0148] Step S803: Based on the task head, output the video-level spatiotemporal feature representation according to the task objective, and perform supervised training on the spatiotemporal joint modeling encoder and the task head based on the output results to obtain the video understanding task model.

[0149] In step S801 of some embodiments, different task heads can be constructed based on different surgical video understanding tasks. The task head, in a deep learning model, is responsible for converting extracted features into a specific task output, typically used for tasks such as classification and regression. The task head primarily processes the features extracted by the backbone network according to the needs of a specific task to generate the final output. For example, in an image classification task, the task head might be a fully connected layer with a Softmax activation function to output the class probability distribution.

[0150] In step S802 of some embodiments, surgical video understanding task data is input into the spatiotemporal joint modeling encoder of the video understanding pre-trained model, and video-level spatiotemporal feature representation is output. By transferring the spatiotemporal representation obtained from pre-training to different types of surgical video analysis tasks, the video understanding pre-trained model, as a unified surgical video basic model, can provide shared video-level spatiotemporal feature support for different tasks.

[0151] In step S803 of some embodiments, the task objective of the task head may include four core tasks: action recognition, surgical stage recognition, triplet recognition, and surgical skill assessment. In the action recognition task, the system receives input surgical videos or video segments and segments the video according to a preset time window to form a continuous sequence of video segments. Each video segment is input into a pre-trained video understanding pre-trained model, where a unified spatiotemporal encoder extracts the corresponding spatiotemporal feature representation, and the action recognition task head outputs the surgical action category within the current time window. Since the action recognition task highly depends on local temporal dynamics and instrument movement patterns, the spatiotemporal joint representation obtained by the video understanding pre-trained model during the pre-training stage can more accurately distinguish different operational behaviors such as grasping, cutting, separating, suturing, and clamping, thereby improving the accuracy and stability of the action recognition results. In the surgical stage recognition task, the system performs continuous temporal modeling on the complete surgical video and outputs the surgical stage category of the current segment based on the video segment-level spatiotemporal features. This task simultaneously relies on instrument and tissue appearance information in a single frame, as well as stage evolution relationships and process context information across time windows. Through large-scale pre-training with surgical videos, the video understanding pre-trained model can learn the phased changes in the surgical procedure, thereby more accurately identifying the boundaries and transitions between different stages and improving its adaptability to complex procedures. In the triplet recognition task, the system identifies structured semantic relationships in the surgical scene based on the input video clips. These structured semantic relationships consist of instruments, actions, and target objects. The unified spatiotemporal encoder first jointly models the changes in instrument states, operational action patterns, and tissue interaction relationships in the video clips, and then the triplet recognition task head outputs the corresponding semantic triplet results. Compared to methods that rely solely on single-frame images for recognition, the video understanding pre-trained model can better distinguish the interaction differences between similar instrument actions and target objects by utilizing continuous dynamic information in the video, thereby improving the semantic integrity and discriminative ability in the triplet recognition task. In the surgical skill assessment task, the system analyzes complete surgical videos or key operation segments, extracts high-level spatiotemporal features reflecting operational proficiency, action stability, process coherence, and temporal coordination, and outputs skill scores or skill levels based on these high-level spatiotemporal features. Because surgical skill assessment relies on both the quality of local movements and a comprehensive understanding of the continuity and standardization of the entire procedure, this task places higher demands on video-level spatiotemporal representation. Video understanding pre-trained models, by simultaneously modeling fine spatial structures and temporal dynamics during the pre-training phase, provide a more comprehensive process representation for skill assessment tasks, thereby improving the ability of assessment results to reflect real-world operational levels.

[0152] One of the above technical solutions has the following advantages or beneficial effects: In the downstream deployment process, this application constructs four types of task heads around a unified spatiotemporal encoder, which are used for action recognition, surgical stage recognition, triplet recognition, and surgical skill evaluation, respectively. Each task shares the same set of pre-trained video-level spatiotemporal feature representations, and different task heads undergo supervised training and result output according to their respective task objectives. By sharing the encoder and separating the task heads, this application supports multiple surgical video understanding tasks on the same model, avoiding the need to train completely independent feature extraction networks for each task, thereby reducing model deployment complexity and improving representation reuse efficiency.

[0153] The solutions of this application embodiment will be described in detail and explained below with reference to specific application examples:

[0154] This application's embodiments can be applied to the field of artificial intelligence technology and can be widely used in computer-aided intervention and intelligent surgical video analysis scenarios. This application proposes a surgical video scene analysis method, device, equipment, and program product. First, a large-scale surgical video dataset is constructed, covering more than 20 surgical operations and more than 10 anatomical structures. Based on the constructed dataset, a video understanding pre-training model is further proposed. This model adopts a reconstruction-based pre-training mechanism, simultaneously modeling spatial structural information and temporal dynamic information within a unified framework to extract video-level spatiotemporal feature representations that possess both anatomical detail perception and temporal process modeling capabilities. Simultaneously, a feature distillation mechanism supervised by an expert model is introduced to enhance the model's learning ability for fine anatomical textures, semantic structures, and high-level dynamic patterns. To verify the effectiveness of this application, a comprehensive evaluation system covering 12 video-level datasets is constructed. This evaluation system covers 5 surgical operations and 4 downstream tasks, including action recognition, surgical stage recognition, triplet recognition, and surgical skill assessment. Experimental results show that the video understanding pre-trained model proposed in this application exhibits superior performance compared to existing pre-trained models in the natural domain and surgical domain. This demonstrates that the present invention can significantly improve the spatiotemporal modeling capability, cross-task transfer capability, and cross-scene generalization capability in surgical video scene understanding, and can provide a unified basic model support for the development and deployment of intelligent surgical systems.

[0155] Please refer to Figure 9. This application embodiment also provides a surgical video scene analysis device, which can implement the above-described surgical video scene analysis method. The device includes:

[0156] Data construction module 901 is used to construct large-scale surgical video datasets;

[0157] The pre-training module 902 is used to input the large-scale surgical video dataset into the video basic model based on spatiotemporal joint modeling for video-level spatiotemporal representation pre-training. Through an adaptive spatiotemporal masking mechanism and a multi-granularity distillation mechanism, spatial structure information and temporal dynamic information are modeled simultaneously to obtain a video understanding pre-trained model.

[0158] The task transfer module 903 is used to acquire surgical video understanding task data and perform task transfer processing on the video understanding pre-trained model to obtain a video understanding task model.

[0159] The scene analysis module 904 is used to input the surgical video to be analyzed into the video understanding task model for surgical scene analysis processing, and to obtain the surgical scene analysis results.

[0160] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0161] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0162] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0163] Please refer to Figure 10, which illustrates the hardware structure of an electronic device according to another embodiment. The electronic device includes:

[0164] The processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0165] The memory 1002 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1002 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001.

[0166] Input / output interface 1003 is used to implement information input and output;

[0167] The communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0168] Bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004);

[0169] The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.

[0170] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0171] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0172] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0173] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0174] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0175] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0176] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0177] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0178] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0179] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0180] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0181] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0182] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0183] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A surgical video scene analysis method, wherein, The method includes the following steps: Construct a large-scale surgical video dataset; The large-scale surgical video dataset is input into the video base model based on spatiotemporal joint modeling for video-level spatiotemporal representation pre-training. Spatial structure information and temporal dynamic information are modeled simultaneously through an adaptive spatiotemporal masking mechanism and a multi-granularity distillation mechanism to obtain a video understanding pre-trained model. The video understanding task data is obtained and the video understanding pre-trained model is subjected to task transfer processing to obtain the video understanding task model. The surgical video to be analyzed is input into the video understanding task model for surgical scene analysis and processing to obtain the surgical scene analysis results.

2. The method according to claim 1, wherein, The construction of the large-scale surgical video dataset includes the following steps: Obtain publicly available surgical video data, and while retaining the original task division and original annotation information, resample and convert the encoding format of the publicly available surgical video data to obtain the first video dataset; Acquire online surgical video data, identify effective segments in the online surgical video data used for surgical scene modeling, and obtain filtered video data; The filtered video data is subjected to video frame visual content detection and audio risk processing to obtain privacy-protected video data; Candidate surgical procedure keywords and anatomical structure keywords are extracted from the privacy-preserving video data, and standardized mapping and label construction are performed on the candidate surgical procedure keywords and anatomical structure keywords based on a pre-constructed vocabulary to obtain labeled video data; Identify and filter duplicate or similar content in the tagged video data to obtain quality control video data; The quality control video data is standardized to obtain a second video dataset; The large-scale surgical video dataset is constructed based on the first video dataset and the second video dataset.

3. The method according to claim 1, wherein, The process of inputting the large-scale surgical video dataset into a video foundation model based on spatiotemporal joint modeling for video-level spatiotemporal representation pre-training, and simultaneously modeling spatial structure information and temporal dynamic information through an adaptive spatiotemporal masking mechanism and a multi-granularity distillation mechanism to obtain a video understanding pre-trained model includes the following steps: Input video segments are selected from the large-scale surgical video dataset and input into the video base model, which includes a spatiotemporal joint modeling encoder, a video reconstruction decoder, and a feature alignment decoder. The input video segment is divided into multiple non-overlapping spatiotemporal blocks arranged in an ordered manner along the time and spatial dimensions, and each spatiotemporal block is mapped to a corresponding visual word. The spatiotemporal joint modeling encoder performs feature extraction and feature update processing on the visual lexical units. Based on the self-attention mechanism, it models the temporal association and spatial neighborhood relationship between the visual lexical units and fuses the temporal context information and spatial context information corresponding to each visual lexical unit to obtain a lexical feature representation containing spatiotemporal dependencies, thereby obtaining enhanced visible lexical units. The input video segment is subjected to adaptive spatiotemporal masking to obtain masked terms; The enhanced visible word and the mask word are concatenated to obtain the concatenated word; The concatenated words are input into the video reconstruction decoder to recover the low-level pixel information of the occluded video area, and the reconstructed video data is output. The concatenated lexical units are input into the feature alignment decoder for spatial feature alignment and multi-granularity distillation, and the output is the aligned feature representation. The loss is calculated based on the reconstructed video data and the alignment feature representation, and the parameters of the video base model are updated based on the calculated loss value to obtain the video understanding pre-trained model.

4. The method according to claim 3, wherein, The adaptive spatiotemporal masking process for the input video segment to obtain masked terms includes the following steps: The motion intensity, instrument activity density, tissue texture complexity, and degree of local scene change of the input video segment are analyzed to obtain the video content complexity; The input video segment is masked by dynamically adjusting the mask position and mask ratio according to the complexity of the video content, thereby obtaining the masked words.

5. The method according to claim 3, wherein, The process of inputting the concatenated lexical units into the feature alignment decoder for spatial feature alignment and multi-granularity distillation, and outputting the aligned feature representation, includes the following steps: The concatenated lexical units are input into the feature alignment decoder, and the output is the prediction space representation. The spatial feature representation of each frame in the input video segment is obtained through an image-level teacher model, and the overall temporal feature representation of the input video segment is obtained through a video-level teacher model. The predicted spatial representation output by the video base model is aligned with the spatial feature representation and the temporal feature representation, respectively. By mapping the predicted features output by the video base model and the features output by the teacher model to the same feature space, and calculating the difference between features based on the loss function, the feature distribution is aligned by minimizing the difference, and the aligned feature representation is obtained.

6. The method according to claim 3, wherein, The process of calculating loss based on the reconstructed video data and the alignment feature representation, and updating the parameters of the video base model based on the calculated loss value to obtain the video understanding pre-trained model includes the following steps: The reconstruction loss is calculated based on the reconstructed video data. Distillation loss is calculated by performing distillation loss on the aligned feature representation based on a preset feature alignment loss function to obtain the distillation loss. By assigning corresponding domain identifiers to surgical video data from different sources in the large-scale surgical video dataset, and performing cross-domain feature matching on video segments with the same surgical procedure category or similar anatomical structures, the feature distribution difference between different domains is calculated based on the maximum mean difference loss or contrastive learning loss, and the domain alignment loss is obtained by minimizing the feature distribution difference. The samples are evaluated based on the predicted loss value or prediction confidence during the training process of the model, and samples with loss values ​​higher than a preset threshold or prediction confidence lower than a preset threshold are identified as difficult samples. The sampling weight or supervision intensity is increased for the difficult samples to obtain the temporal consistency constraint. The parameters of the video base model are updated based on the reconstruction loss, the distillation loss, the domain alignment loss, and the temporal consistency constraint to obtain the video understanding pre-trained model.

7. The method according to any one of claims 1 to 6, wherein, The process of acquiring surgical video understanding task data and performing task transfer processing on the pre-trained video understanding model to obtain a video understanding task model includes the following steps: A task header is constructed based on the surgical video understanding task data; The surgical video understanding task data is input into the spatiotemporal joint modeling encoder of the video understanding pre-trained model, and the output is a video-level spatiotemporal feature representation. Based on the task head, the video-level spatiotemporal feature representation is output according to the task objective. Based on the output results, the spatiotemporal joint modeling encoder and the task head are trained under supervision to obtain the video understanding task model.

8. A surgical video scene analysis device, wherein, The device includes: The data building module is used to construct large-scale surgical video datasets; The pre-training module is used to input the large-scale surgical video dataset into the video base model based on spatiotemporal joint modeling for video-level spatiotemporal representation pre-training. Through an adaptive spatiotemporal masking mechanism and a multi-granularity distillation mechanism, spatial structure information and temporal dynamic information are modeled simultaneously to obtain a video understanding pre-trained model. The task transfer module is used to acquire surgical video understanding task data and perform task transfer processing on the video understanding pre-trained model to obtain a video understanding task model. The scene analysis module is used to input the surgical video to be analyzed into the video understanding task model for surgical scene analysis processing, and to obtain the surgical scene analysis results.

9. An electronic device, wherein, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer program product comprising a computer program, wherein, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.