Video processing method and related device

Abstract

A video processing method, applied to the technical field of artificial intelligence (AI). The video processing method comprises: first dividing a target video into a plurality of video clips, and determining the similarity between an image frame in each video clip and an image frame adjacent thereto. In this way, when the target video is processed by an AI model to obtain a group of feature segments corresponding to each video clip, some feature segments in each group of feature segments are adaptively removed on the basis of the similarity between image frames corresponding to the video clip, thereby achieving the objective of feature reduction. In addition, feature segments in each group are removed in a targeted manner on the basis of the similarity between image frames in each video clip, thereby ensuring video processing accuracy, significantly reducing redundant content to be processed in a video, and improving video processing efficiency.

Description

A video processing method and related apparatus

[0001] This application claims priority to Chinese Patent Application No. 202510297952.2, filed on March 13, 2025, entitled "A Video Processing Method and Related Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence (AI) technology, and in particular to a video processing method and related apparatus. Background Technology

[0003] Large Language Models (LLMs) are deep learning models trained on massive amounts of text data. They can not only generate natural language text, but also deeply understand the meaning of text and handle various natural language tasks, such as text summarization, intelligent question answering, and text translation.

[0004] With the development of artificial intelligence technology, Video Large Language Models (VLM) have made significant progress in video understanding. VLM performs multimodal tasks by fusing a visual encoder with a large language model. However, current VLM suffers from a significant technical bottleneck: it requires a large number of feature fragments to represent a single image. Therefore, VLM often needs to process a large number of feature fragments when processing videos, making processing long videos exceeding several minutes a highly challenging task. Specifically, due to accelerator memory limitations, VLM can typically only process short video clips containing around two hundred images (approximately four minutes), and cannot effectively handle long videos.

[0005] Therefore, there is an urgent need for an optimization scheme for video large language models so that they can effectively process long videos. Summary of the Invention

[0006] This application provides a video processing method and related apparatus, which can improve video processing efficiency.

[0007] Firstly, a video processing method is provided, applicable to scenarios where AI models are used to process videos. This video processing method includes: an execution device first acquiring multiple video segments of a target video, wherein each video segment includes one or more image frames, and the multiple video segments do not have duplicate image frames.

[0008] The execution device then determines the similarity of a first video segment among multiple video clips. The similarity of the first video segment indicates the degree of similarity between image frames in the first video segment and adjacent image frames. A higher similarity of the first video segment indicates a higher degree of similarity between image frames in the first video segment, meaning that the first video segment contains more redundant content.

[0009] Secondly, based on the similarity of the first video segment, the execution device removes some feature segments from the first set of feature segments to obtain the retained feature segments. The retained feature segments include the feature segments that were not removed from the first set of feature segments. The first set of feature segments is obtained by processing the first video segment through an AI model. The retained feature segments are used as inputs into the AI ​​model to perform video processing tasks.

[0010] In this approach, the target video is first divided into multiple video segments, and the similarity between image frames within each video segment and their adjacent frames is determined. Then, when the AI ​​model processes the target video to obtain a set of feature segments for each video segment, some feature segments in each set are adaptively removed based on the similarity of the image frames corresponding to the video segments, achieving feature reduction. Furthermore, by selectively removing each set of feature segments based on the similarity of image frames within the video segments, more feature segments can be removed from video segments with high inter-frame similarity, while fewer feature segments are removed from video segments with low inter-frame similarity. Therefore, while maintaining video processing accuracy, this approach significantly reduces redundant content in the video, improves processing efficiency, and allows the model to process more image frames and more effectively understand long video inputs within the same accelerator memory constraints.

[0011] In one possible implementation, the execution device removes a portion of feature segments from a first set of feature segments based on a first removal ratio. The first removal ratio indicates the ratio between the amount of data in the feature segments to be removed from the first set of feature segments and the total amount of data in the first set of feature segments. The first removal ratio is related to the similarity of the first video segment.

[0012] That is, the higher the similarity of the first video segment, the more redundant content the first video segment contains, and the higher the first elimination ratio corresponding to the first group of feature segments, thus eliminating more feature segments from the first group of feature segments.

[0013] In this scheme, the elimination ratio of video segments is determined based on the similarity of video segments, and then the feature segments corresponding to the video segments are eliminated based on the elimination ratio. This enables the elimination ratio to be adaptively determined based on the redundancy of the content in the video segments, thereby eliminating as many feature segments as possible while ensuring the accuracy of video processing.

[0014] In one possible implementation, the execution device removes some feature segments from a second set of feature segments corresponding to the second video segment based on a second removal ratio among multiple video segments. The similarity between the first and second video segments differs, and the first and second removal ratios differ as well. The retained feature segments also include those not removed from the second set of feature segments.

[0015] In other words, for any two video segments among multiple video segments, if the similarity between the two video segments is not the same, then the elimination ratio of the two video segments will also be different, thereby realizing the adaptive determination of the number of feature segments to be eliminated based on the information redundancy of the video segments.

[0016] In one possible implementation, the execution device determines a first rejection ratio based on the similarity between the first video segment and the similarity between other video segments among a plurality of video segments.

[0017] Specifically, the greater the similarity of the first video segment compared to other video segments, the higher the information redundancy of the first video segment compared to other video segments. Therefore, the first video segment will often have a higher initial elimination ratio.

[0018] In one possible implementation, the execution device determines a first rejection ratio based on the similarity ratio between the similarity of the first video segment and the total similarity; wherein the total similarity is obtained based on the similarity of each of the multiple video segments. For example, the total similarity is the sum of the similarity of each of the multiple video segments.

[0019] In this scheme, the feature segment removal ratio for each video segment is determined based on the ratio between the similarity of a single video segment and the sum of the similarities of all video segments. This enables the feature segment removal ratio for each video segment to be determined by comparing the inter-frame similarity of all video segments, thereby ensuring the accurate removal of redundant information-related feature segments in the target video and improving the accuracy of feature segment removal.

[0020] In one possible implementation, the similarity of the first video segment is used to indicate one or more of the following: the degree of similarity between pixels of an image frame in the first video segment and pixels of adjacent image frames; or, the degree of similarity between vectors corresponding to image frames in the first video segment and vectors corresponding to adjacent image frames; or, the degree of similarity between feature segments corresponding to image frames in the first video segment and feature segments corresponding to adjacent image frames.

[0021] In one possible implementation, the first set of feature fragments includes multiple layers of feature fragments, one of which is extracted by a neural network layer in the AI ​​model.

[0022] The execution device removes some feature segments from the first set of feature segments. Specifically, the execution device removes some feature segments from multiple layers of feature segments, and the amount of data of the feature segments removed from each layer of feature segments is related to the importance of each layer of feature segments. That is, the amount of data of the feature segments removed from each layer of feature segments is not the same, but is determined by the importance of the feature segments in each layer of feature segments.

[0023] In this scheme, for the feature segments corresponding to the same video segment, the feature segments are adaptively removed based on the importance of feature segments at different layers. This can retain the feature segments with higher importance between different layers as much as possible, and remove the feature segments with lower importance. This ensures that the removal of feature segments conforms to the characteristics of feature segments generated by the neural network layers in the model, thereby maximizing the accuracy of video processing after feature segment removal.

[0024] In one possible implementation, the AI ​​model is a video large language model. To determine the importance of feature segments, the execution device acquires cue words corresponding to the target video, which are used to indicate the video processing task corresponding to the target video. Furthermore, the execution device calculates attention scores between the features of each feature segment and the cue words in the multi-layer feature segment hierarchy, where the attention score of one feature segment indicates the importance of that feature segment.

[0025] In this scheme, when the target video has corresponding cue words that indicate the video processing task, since the cue words often indicate the task related to the target video, the attention score of each feature segment in the video segment is calculated based on the features of the cue words. This can more accurately characterize the importance of the feature segments based on the attention scores of the feature segments, thereby ensuring that the feature segments with the least impact on the video processing task are removed, and guaranteeing the accuracy of the final video processing task.

[0026] In one possible implementation, the features of the prompt word comprise multiple feature segments. The execution device obtains the attention score between the features of the first feature segment and the prompt word based on the sum of the attention scores between the first feature segment in the multi-layer feature segment and each feature segment corresponding to the prompt word. For example, the attention score between the features of the first feature segment and the prompt word is simply the sum of the attention scores between the first feature segment and each feature segment corresponding to the prompt word.

[0027] In one possible implementation, to obtain the similarity of a first video segment, the execution device calculates the similarity between each image frame in a plurality of image frames of the first video segment and its adjacent image frames, resulting in multiple similarity scores. The plurality of video segments includes the first video segment. Then, based on the multiple similarity scores, the execution device determines the similarity of the first video segment, for example, by averaging the multiple similarity scores and using the average value as the similarity of the first video segment.

[0028] In one possible implementation, multiple video segments include a second video segment, and the first video segment precedes the second video segment in the target video. When processing the target video, the execution device processes the second video segment using an AI model based on the feature segments retained from the first set of feature segments to obtain the second set of feature segments corresponding to the second video segment.

[0029] In this scheme, the execution device processes multiple video segments sequentially through an AI model. After obtaining a set of feature segments corresponding to a video segment, it performs a culling operation on the set of feature segments to reduce the amount of data in the feature segments and improve the efficiency of processing other video segments based on the generated feature segments.

[0030] In a second aspect, a video processing apparatus is provided, comprising: an acquisition module for acquiring multiple video segments of a target video, wherein each video segment includes one or more image frames; a processing module for determining the similarity of a first video segment among the multiple video segments, wherein the similarity of the first video segment is used to indicate the degree of similarity between an image frame in the first video segment and an adjacent image frame; and a processing module further for removing some feature segments from a first set of feature segments based on the similarity of the first video segment, thereby obtaining retained feature segments, wherein the retained feature segments include feature segments in the first set of feature segments that were not removed, wherein the first set of feature segments is obtained by processing the first video segment using an artificial intelligence (AI) model, and the retained feature segments are used as input into the AI ​​model to perform a video processing task.

[0031] In one possible implementation, the processing module is further configured to: remove some feature segments from the first group of feature segments based on a first removal ratio, wherein the first removal ratio indicates the ratio between the amount of data of the feature segments to be removed from the first group of feature segments and the total amount of data of the first group of feature segments; wherein the first removal ratio is related to the similarity of the first video segment.

[0032] In one possible implementation, the processing module is further configured to: based on a second elimination ratio of the second video segment among multiple video segments, eliminate some feature segments from the second group of feature segments corresponding to the second video segment; wherein the similarity between the first video segment and the second video segment is different, and the first elimination ratio and the second elimination ratio are different, and the retained feature segments also include the feature segments in the second group of feature segments that were not eliminated.

[0033] In one possible implementation, the processing module is further configured to: determine a first elimination ratio based on the similarity between the first video segment and the similarity between other video segments among the multiple video segments.

[0034] In one possible implementation, the processing module is further configured to: determine a first elimination ratio based on the similarity ratio between the similarity of the first video segment and the total similarity; wherein the total similarity is obtained based on the similarity of the multiple video segments respectively.

[0035] In one possible implementation, the similarity of the first video segment is used to indicate one or more of the following: the degree of similarity between pixels of an image frame in the first video segment and pixels of adjacent image frames; or, the degree of similarity between vectors corresponding to image frames in the first video segment and vectors corresponding to adjacent image frames; or, the degree of similarity between feature segments corresponding to image frames in the first video segment and feature segments corresponding to adjacent image frames.

[0036] In one possible implementation, the first set of feature fragments includes multi-layer feature fragments, one layer of which is extracted by a neural network layer in the AI ​​model; the processing module is further configured to: remove some feature fragments from the multi-layer feature fragments, wherein the amount of data of the feature fragments removed from each layer of the multi-layer feature fragments is related to the importance of each layer of feature fragments.

[0037] In one possible implementation, the AI ​​model is a video large language model. The acquisition module is also used to acquire the prompt words corresponding to the target video. The prompt words are used to indicate the video processing task. The processing module is also used to calculate the attention score between the features of each feature segment and the prompt words in the multi-layer feature segments. The attention score of a feature segment is used to indicate the importance of the feature segment.

[0038] In one possible implementation, the features of the prompt word include multiple feature segments; the processing module is further configured to obtain the attention score between the features of the first feature segment and the prompt word based on the sum of the attention scores between the first feature segment in the multi-layer feature segments and each feature segment corresponding to the prompt word.

[0039] In one possible implementation, multiple video segments include a second video segment, and the first video segment is located before the second video segment in the target video; the processing module is further configured to process the second video segment using an AI model based on the feature segments retained in the first set of feature segments to obtain the second set of feature segments corresponding to the second video segment.

[0040] Thirdly, a video processing apparatus is provided, comprising: a processor and a memory; the memory is used to store computer instructions, which, when executed by the processor, cause the video processing apparatus to perform any of the methods described above.

[0041] Fourthly, a computer-readable storage medium is provided that stores instructions which, when executed on a computer, cause the computer to perform the methods of any of the above aspects.

[0042] Fifthly, a computer program product containing instructions is provided, which, when executed on a computer, enable the computer to perform the methods described above.

[0043] In a sixth aspect, a chip is provided, the chip including a processor and a communication interface for communicating with a module other than the chip, the processor for running computer programs or instructions such that a device on which the chip is mounted can perform the methods of any of the above aspects.

[0044] In a seventh aspect, a computing device is provided, the computing device including a video processing apparatus of the third aspect or a chip of the sixth aspect, wherein the video processing apparatus or the chip in the computing device is used to implement the operational steps of the method of any of the above aspects.

[0045] Eighthly, a computing device cluster is provided, comprising at least one computing device, wherein any one computing device is used to run a computer program or instructions, such that the computing device cluster can perform the methods of any of the above aspects. Alternatively, some or all of the computing devices are used together to run a computer program or instructions, such that the computing device cluster can perform the methods of any of the above aspects.

[0046] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods.

[0047] The beneficial effects of aspects two through eight mentioned above can be referred to the introduction of aspect one above, and will not be repeated here. Attached Figure Description

[0048] Figure 1 is a schematic diagram of a system architecture provided in this application;

[0049] Figure 2A is a flowchart illustrating a video processing method provided in this application;

[0050] Figure 2B is a schematic diagram of a feature segment removal method provided in this application;

[0051] Figure 3 is a schematic diagram of a process for removing some feature segments from the first set of feature segments provided in this application;

[0052] Figure 4 is a schematic diagram of a method for removing each group of feature segments based on a removal ratio provided in this application;

[0053] Figure 5 is a schematic diagram of a method for removing feature segments based on the importance of feature segments at different layers, as provided in this application;

[0054] Figure 6 is a schematic diagram of a feature segment removal method based on attention score provided in this application;

[0055] Figure 7 is a schematic diagram of sequential processing of video segments provided in this application;

[0056] Figure 8 is a structural schematic diagram of a video processing device provided in this application;

[0057] Figure 9 is a schematic diagram of the structure of a computing device provided in this application;

[0058] Figure 10 is a schematic diagram of the structure of a computing device cluster provided in this application;

[0059] Figure 11 is a schematic diagram of another computing device cluster provided in this application;

[0060] Figure 12 is a schematic diagram of the structure of a computer-readable storage medium provided in this application. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application are described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some, and not all, of the embodiments of this application. Those skilled in the art will recognize that, with the emergence of new application scenarios, the technical solutions provided by this application are also applicable to similar technical problems.

[0062] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such descriptions can be used interchangeably where appropriate to allow embodiments to be implemented in a sequence other than that illustrated or described in this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. The naming or numbering of steps appearing in this application does not imply that the steps in the method flow must be performed in the chronological / logical order indicated by the naming or numbering. The execution order of named or numbered process steps can be changed according to the desired technical purpose, as long as the same or similar technical effect is achieved. The division of units in this application is a logical division. In practical applications, there may be other division methods. For example, multiple units may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the shown or discussed mutual coupling, direct coupling, or communication connection may be through some interface, and the indirect coupling or communication connection between units may be electrical or other similar forms, none of which are limited in this application. Furthermore, the units or sub-units described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed among multiple circuit units. Some or all of the units can be selected to achieve the purpose of the solution in this application according to actual needs.

[0063] To facilitate understanding, some technical terms used in this application will be introduced below.

[0064] (1) Large Language Model

[0065] Large language models are deep learning models trained on massive amounts of text data that can generate natural language text or understand the meaning of language text. Large language models can handle various natural language tasks, such as text classification, question answering, and dialogue, and are an important pathway to artificial intelligence.

[0066] Specifically, large language models are a technology that has emerged in recent years. Because large language models undergo meticulous data engineering and training processes, their parameters have learned a wealth of existing natural language processing knowledge. This knowledge can now replace humans in many language-related tasks, such as having large language models write code or perform text summarization.

[0067] (2) Transformer network

[0068] Transformer networks are powerful sequence models, but the computation time and memory required increase quadratically with sequence length, significantly increasing the hardware's storage and computing power demands. Essentially, Transformer networks employ a self-attention mechanism. Self-attention is a mechanism that associates different positions within a single sequence to compute a representation of the same sequence, playing a crucial role in machine reading, abstract summarization, and image description generation.

[0069] Taking the Transformer network applied to natural language processing as an example, the Transformer network processes input data of arbitrary length and generates new feature representations of the input data, which are then converted into target words. The self-attention network layer in the Transformer network uses an attention mechanism to capture the relationships between all other words, thereby generating new feature representations for each word. The advantage of the Transformer network is that the attention mechanism can directly capture the relationships between all words in a sentence without considering word positions.

[0070] (3) Video Large Language Model

[0071] Video Big Language Model (VML) is a model based on Big Language Model (ML) that can process both video and text simultaneously. Building upon ML, VML maps each frame of a video to an image embedding vector using a video encoder, aligning the dimensions of the image embedding vectors with those of the text embedding vectors. This allows VML to recognize and process the image embedding vectors, thus completing the video processing.

[0072] (4) Attention Score

[0073] Attention score refers to a score calculated through an attention mechanism, used to measure the relevance and importance between different elements. In the Transformer network, the attention mechanism generates an attention score for each token by calculating the dot product of the query vector, key vector, and value vector. These scores are normalized and become attention weights, used to weight the relevant information of each token.

[0074] (5) Key-Value Cache (KV Cache)

[0075] When using Transformer networks for natural language processing tasks, they typically employ a self-attention mechanism to process the input sequence. In this mechanism, the Transformer network generates a corresponding key (K) vector, value (V) vector, and query (Q) vector for each word in the input sequence. Further, the Transformer network calculates the degree of matching between each query vector and all key vectors, usually achieved through a dot product. Then, using the matching degree between query and key vectors as weights, it calculates a weighted sum of all value vectors to obtain the final result.

[0076] When the Transformer network processes the input sequence, the key vector generated by the Transformer network for each word in the input sequence can be stored in a single matrix, and the value vector generated by the Transformer network for each word in the input sequence can be stored in another matrix. Therefore, KV Cache refers to the key matrix and value matrix generated by the Transformer network for the words in the input sequence, and these key matrices and value matrices are cached.

[0077] (6) Patch Embedding

[0078] Patch embedding is the process of segmenting an image into multiple small image patches and mapping each patch to a high-dimensional vector space. Patch embedding is commonly used in Transformer networks to convert two-dimensional image data into sequential data, facilitating image data processing using the Transformer network.

[0079] The applicant's research revealed a significant technical bottleneck in current video large language models: these models require a large number of feature segments to represent a single image. Therefore, processing videos often involves handling a vast amount of these feature segments, making the processing of long videos exceeding several minutes a highly challenging task. Specifically, current video large language models typically run on accelerators, and the length of feature segments they can effectively process is usually limited. If the video is too long, it will generate too many feature segments, exceeding the accelerator's memory capacity and preventing successful processing. Consequently, current video large language models can generally only process short video segments containing around two hundred images (approximately four minutes), and cannot effectively handle long videos.

[0080] In view of this, this application provides a video processing method that first divides the target video into multiple video segments and determines the similarity between image frames in each video segment and their adjacent image frames. Then, when processing the target video using an AI model to obtain a set of feature segments corresponding to each video segment, some feature segments in each set of feature segments are adaptively removed based on the similarity of the image frames corresponding to each video segment, thereby achieving feature reduction. Furthermore, by selectively removing each set of feature segments based on the similarity of image frames in the video segments, more feature segments can be removed from video segments with high inter-frame similarity, while fewer feature segments can be removed from video segments with low inter-frame similarity. Therefore, while ensuring video processing accuracy, it can significantly reduce redundant content in the video, improve video processing efficiency, and ensure effective processing of long videos.

[0081] Please refer to Figure 1, which is a schematic diagram of a system architecture provided in this application. As shown in Figure 1, in this system architecture, the execution device 10 can be implemented by a single physical host (computing device) or multiple physical hosts (computing device cluster). The execution device 10 includes an accelerator 101 and a processor 102. The accelerator 101 is used to run AI models (such as video large language models) to process inference tasks transmitted by the processor 102. Furthermore, when the accelerator 101 runs the AI ​​model to process the target video, all feature fragments generated by the accelerator 101 are stored in the memory of the accelerator 101. The processor 102 is used to obtain task requests from clients (such as requests to analyze a video) and schedule the accelerator 101 to process the specified inference task based on the task requests from the clients.

[0082] Optionally, the execution device 10 can be used in conjunction with other computing devices, such as data storage devices, load balancers, etc.; the execution device 10 can be deployed on a single physical site or distributed across multiple physical sites.

[0083] In addition, the system architecture also includes a data storage system 11, which is used to store data such as video data, KV cache, or program code.

[0084] Optionally, for persistent data storage, the data storage system 11 can be located external to the execution device 10 and exchange data with the execution device 10 via a network. Alternatively, if the execution device 10 is a physical host, the data storage system 11 can also be located internally to the execution device 10, such as by exchanging data with the processor via a bus. In this case, the data storage system 11 functions as a hard disk. With the data storage system 11, the execution device 10 can use the data in the data storage system 11 (such as a KV cache) or call the program code in the data storage system to implement the video processing method provided in this application.

[0085] Optionally, users can interact with execution device 10 using their respective local devices. For example, a client 121 is deployed on local device 12, and users interact with the execution device through client 121 on local device 12. Local device 12 can represent any computing device, such as a personal computer, computer workstation, smartphone, tablet, laptop, or smart car.

[0086] Local device 12 can interact with execution device 10 through a communication network of any communication mechanism / standard. The communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof.

[0087] Optionally, during the execution of the video processing method by the execution device 10, the local device 12 can provide the execution device 10 with a task request or a video to be processed, so that the execution device 10 can process the video using an AI model to complete the task. Furthermore, after the execution device 10 executes the video processing method and obtains the output result, it can feed the output result back to the local device 12.

[0088] Of course, in some possible examples, execution device 10 and local device 12 can also be the same device, that is, the client can be directly deployed on execution device 10 to obtain the user's task request.

[0089] Please refer to Figure 2A, which is a flowchart illustrating a video processing method provided in this application. As shown in Figure 2A, the video processing method includes the following steps 201-203.

[0090] Step 201: Obtain multiple video segments of the target video, where each video segment includes one or more image frames.

[0091] In this application, the target video is a video that needs to be processed by an AI model, and the target video includes multiple ordered image frames. For example, the target video can be a long video with a duration of 5 minutes, 10 minutes, or 30 minutes, and the number of image frames included in the target video can be, for example, 300, 600, or 1800. This application does not limit the duration of the target video or the number of image frames included.

[0092] For the execution device, it can acquire the target video by obtaining a user's task request. For example, the user sends a task request to the execution device through a local device, carrying the target video and requesting processing of it. The target video can also be pre-stored in a database, in which case the execution device can retrieve it from the database. In general, this application does not limit the specific method by which the execution device acquires the target video.

[0093] After acquiring the target video, the execution device can segment the target video into multiple ordered video segments. Each video segment includes all the image frames from the target video, and there are no duplicate image frames between the video segments (i.e., adjacent video segments do not overlap in time). Furthermore, each video segment can include one or more image frames. The number of image frames in different video segments can be the same or different. For example, assuming the target video contains 300 image frames, the execution device can divide it into 10 video segments with a length of 30 image frames each, resulting in 10 video segments, each containing 30 consecutive image frames. In this case, the 10 video segments contain the same number of image frames. Alternatively, for the target video containing 300 image frames, a non-fixed-length segmentation method can be used. For example, the first video segment could include 20 image frames (frames 1 to 20), the second video segment could include 40 image frames (frames 21 to 60), and so on.

[0094] It should be understood that in this step, the target video can also be divided into multiple video segments by other devices, and the executing device can directly obtain the pre-divided multiple video segments of the target video from the other devices.

[0095] Step 202: Determine the similarity of a first video segment among multiple video segments. The similarity of the first video segment is used to indicate the degree of similarity between image frames in the first video segment and adjacent image frames.

[0096] After acquiring multiple video segments, the execution device can calculate the similarity of each video segment to determine the degree of similarity between image frames in a video segment and their adjacent image frames (i.e., inter-frame similarity). Specifically, if the similarity between image frames in a video segment is high, meaning the video segment contains more redundant content, then the similarity of the video segment is higher; conversely, if the similarity between image frames in a video segment is low, meaning the video segment contains less redundant content, then the similarity of the video segment is lower.

[0097] For example, for a first video segment among multiple video segments, the execution device can calculate the similarity between each image frame in the multiple image frames of the first video segment and its adjacent image frames, thus obtaining multiple similarities. The first video segment can be any one of the multiple video segments. The adjacent image frames of the first image frame in the first video segment can be the N nearest frames of the first image frame, where N is a positive integer. The value of N can be determined according to actual needs, and this application does not limit it. For example, if N equals 1, the adjacent image frames of the first image frame can be the preceding or following image frame of the first image frame. If N equals 2, the adjacent image frames of the first video frame can include the two image frames before and after the first video frame. For example, when the sequence number of the first video frame is 15, the adjacent image frames of the first video frame include: frame 13, frame 14, frame 16, and frame 17. Therefore, for each image frame in the first video segment, a corresponding similarity can be calculated, thus obtaining multiple similarities corresponding one-to-one with the multiple image frames. It should be noted that if the adjacent image frames of an image frame are defined as including multiple image frames (i.e., N is greater than 1 as mentioned above), then the similarity of an image frame can be the average of the similarity between that image frame and each of its adjacent image frames.

[0098] Then, based on multiple similarity scores, the execution device determines the similarity score corresponding to the first video segment. For example, the execution device calculates the average of multiple similarity scores to obtain the similarity score corresponding to the first video segment. Currently, other methods can also be used, and this application is not limited to them.

[0099] It's important to note that for multiple video clips, since the similarity of a video clip is calculated by comparing the similarity between an image frame and its adjacent frames, calculating the similarity between certain video clips may involve calculating similarity across multiple clips. For example, when calculating the similarity between an image frame in the first video clip and its adjacent frames, it's necessary to calculate the similarity between the image frame and the two preceding and following frames. Therefore, for the first image frame in the first video clip, the similarity between the first image frame of the first video clip and the last image frame of the preceding video clip needs to be calculated; similarly, for the last image frame of the first video clip, the similarity between the last image frame of the first video clip and the first image frame of the following video clip needs to be calculated. Conversely, if the first video clip contains only one image frame, then the similarity between the image frame in the first video clip and the last image frame of the preceding video clip, as well as the similarity between the image frame in the first video clip and the first image frame of the following video clip, needs to be calculated.

[0100] Of course, besides determining the similarity of a video segment by calculating the similarity between each image frame in the video segment and its adjacent image frames, the execution device can also determine the similarity of the video segment by combining adjacent image frames pairwise. This application does not specifically limit how the execution device calculates the similarity of the video segments. Optionally, the similarity of the first video segment is used to indicate one or more of the following: the degree of similarity between pixels of an image frame in the first video segment and pixels of adjacent image frames; or, the degree of similarity between vectors corresponding to image frames in the first video segment and vectors corresponding to adjacent image frames; or, the degree of similarity between feature segments corresponding to image frames in the first video segment and feature segments corresponding to adjacent image frames.

[0101] The similarity between pixels in an image frame of the first video segment and pixels in adjacent image frames can be determined by the similarity of pixels at the same or adjacent positions in the image frame and adjacent image frames. Specifically, for a given image frame in the first video segment, the average or sum of the differences between pixels at each position in that image frame and adjacent image frames can be calculated to obtain the similarity of pixels at the same position in the image frame and adjacent image frames. The larger the average or sum of the differences between pixels at each position in the image frame and adjacent image frames, the lower the pixel similarity between the image frame and adjacent image frames; conversely, the smaller the average or sum of the differences between pixels at each position in the image frame and adjacent image frames, the higher the pixel similarity between the image frame and adjacent image frames. Thus, based on the pixel similarity between each image frame in the first video segment and adjacent image frames, the similarity between pixels in the image frame of the first video segment and pixels in adjacent image frames can be determined.

[0102] In this implementation, visual changes are evaluated by directly comparing the similarity of pixels between image frames. This makes the calculation of the similarity of image frames conform to the intuition of human visual perception, ensuring that feature segments that are similar in human visual perception can be removed as much as possible, thereby improving the accuracy of feature segment removal.

[0103] The similarity between the vectors corresponding to image frames in the first video segment and the vectors corresponding to adjacent image frames refers to determining the similarity of the first video segment by comparing the similarity between the vectors corresponding to the image frames. That is, for a specific image frame in the first video segment, the similarity between the vector of that image frame and the vectors of adjacent image frames can be calculated. Thus, based on the similarity between the vectors of each image frame in the first video segment and the vectors of adjacent image frames, the similarity between the vectors corresponding to the image frames in the first video segment and the vectors corresponding to the adjacent image frames can be determined. The vectors corresponding to the image frames can be extracted using neural network models such as visual encoders. The visual encoder can be, for example, a Transformer network or a Convolutional Neural Network (CNN), and this application does not limit the specific structure of the visual encoder.

[0104] Specifically, since vector representations are more robust and can more stably reflect the similarity of image content, and are less sensitive to noise and small changes, similarity calculations can better capture the semantic information of images by converting image frames into vectors. This makes them more reliable in semantic similarity evaluation and improves the accuracy of feature segment removal.

[0105] When the AI ​​model used to process the target video is a visual large language model, the execution device can use the visual encoder in the visual large language model to extract the vector of each image frame in each video segment, and then obtain the similarity of each image frame by calculating the similarity between the vector of each image frame and the vectors of adjacent image frames. The method for calculating the similarity between the vector of each image frame and the vectors of adjacent image frames can be, for example, cosine similarity or Euclidean distance, and this application does not specifically limit this method.

[0106] The similarity between the feature segments corresponding to the image frames in the first video segment and the feature segments corresponding to the adjacent image frames refers to determining the similarity of the first video segment by comparing the similarity between the feature segments corresponding to the image frames (e.g., cosine similarity or Euclidean distance).

[0107] In this implementation, while maintaining the advantages of semantic similarity calculation, the assumption that feature segments of adjacent image frames are similar is introduced, and the similarity of image frames is evaluated by comparing feature segments. This design not only has strong semantic understanding capabilities, but also significantly reduces computational complexity through the comparison of local features, improving computational efficiency while ensuring accuracy, making it particularly suitable for processing continuous frames.

[0108] Step 203: Based on the similarity of the first video segment, some feature segments are removed from the first set of feature segments to obtain the retained feature segments. The retained feature segments include the feature segments that were not removed from the first set of feature segments. The first set of feature segments is obtained by processing the first video segment through an artificial intelligence (AI) model. The retained feature segments are used as inputs into the AI ​​model to perform video processing tasks.

[0109] After obtaining the target video, the execution device can further process the target video using an AI model to extract multiple sets of feature segments. These multiple sets of feature segments can correspond one-to-one with multiple video segments in the target video, meaning each video segment corresponds to one set of feature segments. For example, the first video segment among multiple video segments corresponds to the first set of feature segments in the multiple sets of feature segments.

[0110] Specifically, when the AI ​​model is a video-based large language model, multiple feature segments can include a key-value cache. When processing the target video using the AI ​​model, the execution device employs patch embedding to segment each image frame in the target video into multiple small image patches, mapping each image patch to an image embedding vector. This results in a sequence of data where each image frame is mapped to multiple image embedding vectors. Thus, by inputting the sequence data corresponding to each image frame in a video segment into the AI ​​model, a set of feature segments extracted by the AI ​​model for each video segment's sequence data can be obtained. This set of feature segments can, for example, include multiple feature segments, with each feature segment corresponding to an image patch within an image frame.

[0111] For example, when the AI ​​model is a video large language model, a set of feature segments includes multiple feature segments, and each feature segment can include a key matrix, a value matrix, and a query matrix.

[0112] After processing each video segment using the target video, the execution device obtains a set of feature segments corresponding to each video segment. At this point, the execution device can remove some feature segments from each set of feature segments based on the similarity of each video segment, thus obtaining the retained feature segments. Specifically, the retained feature segments are obtained by removing some feature segments from each of the multiple sets of feature segments.

[0113] For example, for the first video segment among multiple video segments, the execution device removes some feature segments from the first group of feature segments based on the similarity of the first video segment, wherein the feature segments that are not removed from the first group of feature segments are included in the aforementioned retained feature segments.

[0114] Specifically, the multiple feature segments obtained by the processing device from the target video are actually all feature segments extracted by the AI ​​model from multiple video segments, which can easily lead to information redundancy. Furthermore, the information redundancy in most videos is not uniformly distributed but exhibits significant dynamism as the video content changes. For example, suppose the target video is an outdoor video, with the first half focusing on static outdoor scenery and the second half on dynamic people. In this case, the first half of the video shows little change (i.e., high similarity between image frames), resulting in significant information redundancy; while the second half shows significant change (i.e., low similarity between image frames), resulting in less information redundancy.

[0115] Therefore, considering the non-uniform distribution of information redundancy in videos, this application determines how to remove feature segments from a video segment based on the similarity between image frames within the video segment. If the similarity of a video segment is high, it means that the image frames in the video segment are relatively similar (i.e., the video segment has a lot of redundant information). Therefore, the execution device can remove more feature segments from the set of feature segments corresponding to the video segment, thereby minimizing the number of feature segments that need to be processed. If the similarity of a video segment is low, it means that the image frames in the video segment are less similar (i.e., the video segment has less redundant information). Therefore, the execution device can remove fewer feature segments from the set of feature segments corresponding to the video segment, thereby preserving as much of the key content of the video segment as possible.

[0116] For example, the amount of data in the feature segments removed from the first set of feature segments is related to the similarity to the first video segment. For instance, there is a positive correlation between the amount of data in the feature segments removed from the first set of feature segments and the similarity to the first video segment. That is, the higher the similarity to the first video segment, the larger the amount of data in the feature segments removed from the first set of feature segments; conversely, the lower the similarity to the first video segment, the smaller the amount of data in the feature segments removed from the first set of feature segments.

[0117] Specifically, please refer to Figure 2B, which is a schematic diagram of a feature segment removal method provided in this application. As shown in Figure 2B, assume the target video is divided into N video segments, namely video segment 1 to video segment N, and each video segment includes the same number of image frames. The execution device calculates the corresponding similarity for each video segment from 1 to N, i.e., similarity 1 to similarity N. After processing the target video using an AI model, N sets of feature segments are obtained, namely the first set of feature segments to the Nth set of feature segments. Each set of feature segments corresponds to a video segment. For example, the first set of feature segments corresponds to video segment 1, the second set of feature segments corresponds to video segment 2, and so on, until the Nth set of feature segments corresponds to video segment N. For the obtained N sets of feature segments, the execution device can remove some feature segments based on the similarity corresponding to each video segment. For example, in Figure 2B, video segments 1, 2, and 4 have high similarity, therefore, more feature segments are removed from the first, second, and fourth sets of feature segments. Video segment 3 and video segment N have low similarity, so fewer feature segments were removed from the third and Nth feature segments.

[0118] Please refer to Figure 3, which is a flowchart illustrating the process of removing some feature segments from the first group of feature segments provided in this application. As shown in Figure 3, step 203 above, which removes some feature segments from the first group of feature segments based on the similarity of the first video segment, can be performed according to steps 2031-2032 below. It should be noted that steps 2031-2032 describe how to remove some feature segments from the first group of feature segments corresponding to the first video segment. In practical applications, for each group of feature segments corresponding to each of the multiple video segments, the execution device can refer to steps 2031-2032 to remove some feature segments from each group of feature segments.

[0119] Step 2031: Based on the similarity of the first video segment, determine the first elimination ratio corresponding to the first video segment.

[0120] Specifically, since the image frames included in each video segment may be different, and the amount of data of a set of feature segments corresponding to a video segment is often not fixed under different video segment division methods, this application can determine the elimination ratio based on similarity, and then eliminate feature segments based on the elimination ratio.

[0121] For example, the execution device may remove some feature segments from the first group of feature segments based on a first removal ratio, wherein the first removal ratio is used to indicate the ratio between the amount of data of the feature segments to be removed from the first group of feature segments and the total amount of data of the first group of feature segments; wherein the first removal ratio is related to the similarity of the first video segment.

[0122] Simply put, the higher the similarity of the first video segment, the higher the first elimination ratio of the first set of feature segments, and thus more feature segments are eliminated from the first set of feature segments.

[0123] For example, the execution device removes some feature segments from the second set of feature segments corresponding to the second video segment based on a second removal ratio among multiple video segments. The similarity between the first and second video segments is different, and the first and second removal ratios are different. The retained feature segments also include those not removed from the second set of feature segments.

[0124] In other words, for any two video segments among multiple video segments, if the similarity between the two video segments is not the same, then the elimination ratio of the two video segments will also be different, thereby realizing the adaptive determination of the number of feature segments to be eliminated based on the information redundancy of the video segments.

[0125] Optionally, in order to determine the elimination ratio corresponding to the first set of feature segments, the execution device may determine the first elimination ratio based on the similarity between the first video segment and the similarity between other video segments in the multiple video segments.

[0126] Generally speaking, the greater the similarity of the first video segment compared to other video segments, the higher the information redundancy of the first video segment, and therefore the higher the initial elimination ratio of the first video segment. Conversely, the smaller the similarity of the first video segment compared to other video segments, the lower the information redundancy of the first video segment, and therefore the lower the initial elimination ratio of the first video segment.

[0127] For example, the execution device may first determine the similarity ratio between the similarity of the first set of feature segments and the total similarity, where the total similarity is obtained based on the similarity of each of the multiple video segments. For instance, the total similarity is the sum of the similarities of each of the multiple video segments.

[0128] Then, the execution device determines the elimination ratio corresponding to the first group of feature segments based on the obtained similarity ratio. For example, the execution device uses the similarity ratio as the elimination ratio corresponding to the first group of feature segments. Alternatively, the execution device combines the maximum amount of feature segments that the accelerator memory can store, the total amount of data of all feature segments corresponding to the target video, and the similarity ratio to determine the elimination ratio corresponding to the first group of feature segments, to ensure that all the ultimately retained feature segments can be stored in the accelerator memory.

[0129] In this scheme, the feature segment removal ratio for each video segment is determined based on the ratio between the similarity of a single video segment and the sum of the similarities of all video segments. This enables the feature segment removal ratio for each video segment to be determined by comparing the inter-frame similarity of all video segments, thereby ensuring the accurate removal of redundant information-related feature segments in the target video and improving the accuracy of feature segment removal.

[0130] For example, please refer to Figure 4, which is a schematic diagram of removing each group of feature segments based on a removal ratio provided by this application. As shown in Figure 4, the total similarity can be obtained based on the similarity 1-similarity N corresponding to video segments 1-N respectively. For example, the total similarity is the sum of similarity 1-similarity N. Then, based on the similarity corresponding to each video segment and the total similarity, the removal ratio corresponding to each video segment can be determined, thereby realizing the removal of feature segments based on the removal ratio corresponding to each video segment. For example, the removal ratio 1 corresponding to video segment 1 is the ratio of similarity 1 to the total similarity; the removal ratio 2 corresponding to video segment 2 is the ratio of similarity 2 to the total similarity... and the removal ratio N corresponding to video segment N is the ratio of similarity N to the total similarity.

[0131] Step 2032: Based on the first elimination ratio corresponding to the first video segment, some feature segments in the first group of video segments are eliminated.

[0132] After obtaining the first rejection ratio corresponding to the first video segment, the execution device can determine the amount of data of the feature segments to be rejected in the first group of video segments based on the total number of feature segments in the first group of video segments and the first rejection ratio. Similarly, the execution device can also determine the amount of data of the feature segments to be rejected in the second group of video segments corresponding to the second video segment based on the rejection ratio of the second video segment. After determining the total amount of data of the feature segments to be rejected for each video segment, the feature segments to be rejected can be rejected in the following two ways.

[0133] Method 1: Adaptively (dynamically) remove feature fragments extracted from each layer of the neural network of the AI ​​model.

[0134] The applicant's research revealed that during the processing of video clips using an AI model, each neural network layer generates corresponding feature fragments, and these feature fragments need to be retained for subsequent video processing tasks. However, the redundancy of feature fragments generated by different neural network layers is not uniform.

[0135] Specifically, the degree of information aggregation varies across different neural network layers in an AI model. In shallower neural network layers, the degree of information aggregation is lower, so the importance of feature fragments generated by the same shallower neural network layer is not significantly different. In deeper neural network layers, the degree of information aggregation is higher (i.e., important information is concentrated in some feature fragments), so the importance of feature fragments generated by the same deeper neural network layer is significantly different.

[0136] Based on this, for a feature segment corresponding to a video segment, this application removes feature segments with different amounts of data from the feature segments extracted from different neural network layers in the AI ​​model, thereby achieving adaptive removal of feature segments based on different inter-layer redundancy.

[0137] For example, the target video includes a first set of feature segments among multiple sets of feature segments. This first set of feature segments comprises multiple layers of feature segments, with each layer extracted by a neural network layer in the AI ​​model. In other words, for a given video segment, the first set of feature segments includes multiple layers of feature segments, and each layer is extracted by a neural network layer in the AI ​​model. That is, each layer of feature segments corresponds one-to-one with a different neural network layer in the AI ​​model.

[0138] In this context, the amount of data in the feature segments removed from each layer of a multi-layer feature segment is related to the importance of the feature segments within that layer. That is, the amount of data removed from each layer is not the same, but rather determined by the importance of the feature segments within that layer. The importance of a feature segment can refer to its impact on video processing accuracy. The higher the importance of a feature segment, the greater its impact on video processing accuracy (i.e., the greater the impact on accuracy after its removal); conversely, the lower the importance of a feature segment, the smaller its impact on accuracy (i.e., the smaller the impact on accuracy after its removal).

[0139] Generally, in AI models, the amount of data extracted from feature segments by each neural network layer is the same. Therefore, in a multi-layer feature segment system, if a certain layer has more important feature segments (i.e., the higher the importance of the feature segments), then the amount of data removed from the feature segments in that layer will be smaller; conversely, if a certain layer has fewer important feature segments (i.e., the lower the importance of the feature segments), then the amount of data removed from the feature segments in that layer will be larger.

[0140] For example, please refer to Figure 5, which is a schematic diagram of a method for removing feature segments based on the importance of feature segments at different layers, as provided in this application. As shown in Figure 5, after processing video segment 1 using an AI model, a first group of feature segments is obtained. The AI ​​model includes multiple neural network layers, namely neural network layer 1 to neural network layer M. Each neural network layer outputs one layer of feature segments after processing video segment 1, thus forming the first group of feature segments composed of multiple layers of feature segments. For example, neural network layer 1 outputs the first layer of feature segments, neural network layer 2 outputs the second layer of feature segments, and so on, neural network layer M outputs the Mth layer of feature segments; and the first layer of feature segments to the Mth layer of feature segments constitute a group of feature segments (i.e., the aforementioned first group of feature segments). For the first group of feature segments, the execution device can uniformly determine the feature segments to be removed based on the importance of each feature segment, rather than independently removing each layer of feature segments.

[0141] In this solution, for feature segments corresponding to the same video clip, feature segments are adaptively removed based on the importance of feature segments at different layers. This approach aims to retain as many high-importance feature segments as possible while removing low-importance ones, ensuring that the removal of feature segments conforms to the characteristics of feature segments generated by the neural network layers in the model, thereby maximizing the accuracy of video processing after feature segment removal. For example, to determine the importance of a feature segment, the execution device can calculate the attention score of each feature segment in the multi-layer feature segment hierarchy, where the attention score indicates the importance of the feature segment. For instance, a feature segment might include a Query matrix, a Key matrix, and a Value matrix; the execution device could then generate the attention score corresponding to the feature segment by calculating the dot product of the Query matrix and the Key matrix. Generally, a higher attention score indicates a higher importance of the feature segment.

[0142] Then, the execution device removes some feature segments from the multi-layer feature segments based on the attention score of each feature segment. For example, the execution device can sort the feature segments in the multi-layer feature segments in descending order of attention score, thereby removing the feature segments with the lowest attention scores. It should be noted that since the multi-layer feature segments belong to the same group of feature segments, after determining the removal ratio of a group of feature segments through the aforementioned embodiments, the multi-layer feature segments in this group of feature segments can be removed based on that removal ratio.

[0143] It should be noted that when using the attention score of each feature segment to remove some feature segments from multiple layers, there may be instances where the actual removal ratio of one or more feature segments is 0, meaning that the entire layer of feature segments is not removed. Of course, in some examples, the execution device can set a maximum removal ratio for each layer of feature segments, meaning the proportion of feature segments removed from each layer cannot exceed the maximum removal ratio. This ensures that each layer retains some feature segments, preventing the complete removal of feature segments extracted by certain neural network layers due to relatively low attention scores.

[0144] For example, please refer to Figure 6, which is a schematic diagram of a feature segment removal method based on attention score provided in this application. As shown in Figure 6, after processing video segment 1 through an AI model, a first group of feature segments is obtained. Each of the neural network layers 1 to M in the AI ​​model outputs a layer of feature segments, thus forming a first group of feature segments composed of multiple layers of feature segments. The first group of feature segments includes feature segments from layer 1 to layer M. Then, the execution device calculates the attention score of each feature segment in the first group of feature segments and sorts all feature segments in the first group of feature segments according to the attention scores from high to low or from low to high. Finally, the execution device removes the feature segments with the lowest attention scores based on the removal ratio 1 corresponding to the first group of feature segments. For example, assuming the removal ratio 1 is 50%, the execution device removes 50% of the feature segments with the lowest attention scores from the first group of feature segments.

[0145] Optionally, when the AI ​​model is a video large language model, the execution device can acquire the prompt words corresponding to the target video. These prompt words are used to indicate the video processing task to be performed on the target video. Furthermore, the execution device extracts features from the prompt words using a visual large language model to obtain the features of the prompt words. For example, the prompt word could be "Please identify the identity of the user appearing in the video," meaning the video processing task indicated by the prompt word is a face recognition task; or, for example, the prompt word could be "Please detect the non-motorized vehicle appearing in the video," meaning the video processing task indicated by the prompt word is an object detection task.

[0146] Thus, when calculating the attention score of each feature segment within a multi-layer feature set, the execution device can calculate the attention score between each feature segment and the features of the cue word. For example, a feature segment can include a Query matrix, a Key matrix, and a Value matrix, and the features of the cue word extracted by the visual large language model can also include a Query matrix, a Key matrix, and a Value matrix. The execution device can generate the attention score corresponding to the feature segment by calculating the dot product between the Query matrix of the cue word's features and the Key matrix of the feature segment of the video segment. Generally, the higher the attention score of a feature segment, the higher its importance.

[0147] Specifically, since prompt words often include multiple words, their features consist of multiple feature segments. In this case, for the first feature segment in the multi-layered feature segment structure, the execution device can obtain the attention score between the first feature segment and the features of the prompt word based on the sum of the attention scores between the first feature segment and each feature segment corresponding to the prompt word. For example, the attention score between the first feature segment and the features of the prompt word is simply the sum of the attention scores between the first feature segment and each feature segment corresponding to the prompt word.

[0148] In this approach, when the target video has corresponding cue words that indicate the video processing task, since the cue words often indicate the task related to the target video, calculating the attention score of each feature segment in the video segment based on the features of the cue words can more accurately characterize the importance of the feature segments based on their attention scores. This ensures that the feature segments with the least impact on the video processing task are removed, thus guaranteeing the accuracy of the final video processing task.

[0149] Method 2: Uniformly remove feature fragments extracted from each layer of the neural network of the AI ​​model.

[0150] In some possible examples, the execution device may also allocate the same elimination ratio or amount of data to be eliminated for each layer of feature segments corresponding to each video segment based on the elimination ratio corresponding to each video segment or the total amount of feature segments to be eliminated (i.e., uniform distribution). Here, each layer of feature segments corresponding to a video segment refers to the feature segments extracted by each neural network layer of the AI ​​model for that video segment. In this approach, it is not necessary to consider the importance of feature segments in each layer of the AI ​​model's neural network and eliminate the same proportion or amount of feature segments from each layer. It should be understood that because this approach does not require calculating the importance of feature segments (attention scores) in each layer of the neural network, the computational cost is lower.

[0151] Understandably, when the AI ​​model used to process the target video is a visual large language model, in order to ensure that the AI ​​model can perceive the temporal order of different video segments in the target video, the AI ​​model often needs to process subsequent video segments based on the feature fragments of the already processed video segments. In other words, for any video segment in the target video other than the first video segment, when processing that video segment by the AI ​​model, it often needs to rely on the feature fragments of all the video segments preceding that video segment.

[0152] Based on this, in this application, the execution device can process multiple video segments sequentially through an AI model, and after obtaining a set of feature segments corresponding to a video segment, perform a culling operation on the set of feature segments to reduce the amount of data of the feature segments and improve the efficiency of processing other video segments based on the generated feature segments.

[0153] For example, the multiple video segments obtained based on the target video segmentation include a first video segment and a second video segment, and the first video segment is located before the second video segment in the target video.

[0154] The execution device first processes the first video segment using an AI model to obtain the first set of feature segments corresponding to the first video segment. Then, the execution device removes some feature segments from the first set of feature segments, obtaining the retained feature segments from the first set of feature segments. These retained feature segments are obtained after removing some feature segments from the first set of feature segments corresponding to the first video segment. In other words, after processing the first video segment using the AI ​​model, the execution device removes feature segments from the resulting set of feature segments corresponding to the first video segment.

[0155] Secondly, the execution device processes the second video segment using an AI model based on the feature segments retained in the first set of feature segments to obtain the second set of feature segments corresponding to the second video segment.

[0156] In other words, when the execution device processes the second video segment through the AI ​​model, it no longer relies on all the feature segments corresponding to the first video segment, but on the feature segments retained in the first group of feature segments corresponding to the first video segment. This reduces the amount of feature segment data that the AI ​​model needs to process and improves the efficiency of the AI ​​model in processing the second video segment.

[0157] It should be noted that if there are other video segments before the first video segment, the execution device, when processing the second video segment using the AI ​​model, also needs to perform the processing of the second video segment based on the preserved feature segments corresponding to the other video segments.

[0158] For example, please refer to Figure 7, which is a schematic diagram of sequential processing of video segments provided by this application. As shown in Figure 7, assume that the first three video segments in the multiple video segments divided into the target video are video segment 1 to video segment 3. The execution device first processes video segment 1 through an AI model to obtain the first group of feature segments corresponding to video segment 1, and then removes the first group of feature segments based on the obtained removal ratio 1, thereby obtaining the feature segments retained in the first group of feature segments.

[0159] Then, the execution device continues to process video segment 2 based on the retained feature segments in the first group of feature segments using an AI model to obtain the second group of feature segments corresponding to video segment 2, and removes the second group of feature segments based on the obtained removal ratio 2 to obtain the retained feature segments in the second group of feature segments.

[0160] Secondly, the execution device continues to process video segment 3 through an AI model based on the retained feature segments in the first group of feature segments and the retained feature segments in the second group of feature segments to obtain the third group of feature segments corresponding to video segment 3. Based on the obtained elimination ratio 3, the third group of feature segments is eliminated to obtain the retained feature segments in the third group of feature segments.

[0161] Similarly, when processing any subsequent video segment, the execution device needs to perform video segment processing based on the retained feature segments corresponding to the already processed video segments.

[0162] The above describes the process of removing feature segments generated by the AI ​​model to compress the feature segments during the processing of the target video using the AI ​​model.

[0163] Furthermore, in this application, the retained feature segments obtained after the execution device removes some feature segments from multiple sets of feature segments are used as features of the target video to perform video processing tasks. That is, when performing corresponding video processing tasks on the target video, the processing is no longer based on multiple sets of feature segments corresponding to the target video, but on the retained feature segments. Video processing tasks can be, for example, video analysis tasks, video question answering tasks, video enhancement tasks, and video generation tasks. Among these, a video question answering task can be a task that answers a user-provided question text in conjunction with the target video. That is, the user can simultaneously provide the target video and question text, requesting the execution device to answer the question indicated by the question text in conjunction with the target video. In this case, the execution device can process the target video and question text simultaneously using a video large language model to obtain the answer text. Where the target video is a visual large language model, the question text can be provided through prompt words.

[0164] For example, in a possible application scenario where the target video is from a security or traffic scenario, the video processing task would be a video analysis task. Specifically, the video analysis task involves analyzing the target video to determine whether specific behaviors (such as user violations) occur within it.

[0165] Of course, video processing tasks can also be other types of tasks. The video processing method provided in this application can be applied to various video processing scenarios. This application does not limit the specific scenario in which the video processing method is applied.

[0166] For example, after obtaining the retained feature segments, the execution device can continue to process the retained feature segments using an AI model to obtain a video processing result. This video processing result can be, for example, a video analysis result, a video question-and-answer result, a video enhancement result, or a generated video. That is, after extracting all feature segments corresponding to the target video, the AI ​​model does not perform further processing on all feature segments. Instead, it first removes some feature segments from all feature segments and then uses the AI ​​model to continue processing the remaining feature segments.

[0167] In the case of a visual large language model, the process by which the execution device uses the AI ​​model to process the target video and obtain multiple sets of feature segments is called the pre-filling stage, while the process by which the execution device continues to process the retained feature segments through the AI ​​model is called the generation stage.

[0168] The above describes the execution flow of the video processing method provided in this application. The following will describe the execution process of the video processing method in detail with specific examples.

[0169] Specifically, for a given target video, the execution device can first divide the target video into multiple video segments of equal length, and dynamically assign different compression ratios (i.e., feature segment removal ratios) to each video segment based on the inter-frame similarity of each video segment, thereby flexibly reducing redundant information in the video's temporal dimension. Furthermore, for the feature segments generated by different neural network layers within each video segment, different compression ratios are assigned to different neural network layers based on the importance of the feature segments generated by each neural network layer, thereby further reducing inter-layer redundancy.

[0170] In practical applications, the process by which the execution device processes the target video may include the following multiple stages.

[0171] Phase 1: Input preprocessing.

[0172] In Phase 1, the user provides the target video to be processed and cue words describing the video processing task. The execution device can then process the target video using a visual encoder within a visual large language model, thereby converting the target video into video features. Where T is the total number of image frames in the target video, N is the number of image patches divided into for each image frame during processing, and d is the feature dimension. Furthermore, the execution device processes the cue words through the word embedding layer in the visual large language model, thereby generating cue word features. Where L is the length of the prompt word.

[0173] Phase Two: Segment the video.

[0174] For a given target video, the execution device needs to segment the target video, dividing it into multiple video segments of equal length, each segment containing τ image frames. The video features of a single video segment can be represented as M. i Therefore, the video features M of the target video can be represented based on the following formula 1.

[0175] Phase 3: Determine the proportion of video segments to be removed.

[0176] Specifically, after dividing the video into multiple segments, the execution device can determine the rejection ratio for each video segment. Determining the rejection ratio for each video segment involves the following three steps.

[0177] Step 1: Calculate the inter-frame similarity in the video clip.

[0178] For a given video segment, the execution device needs to calculate the similarity between each image frame in the video segment and its adjacent image frames to obtain a similarity score for each image frame.

[0179] For example, for the video features of the i-th video segment The execution device can calculate the distance between adjacent image frames. Specifically, as shown in Formula 2 below.

[0180] Where, d i [t] represents the similarity between the t-th and (t+1)-th image frames in the i-th video segment, and Sim(·) represents the cosine similarity.

[0181] Step 2: Calculate the average similarity of image frames in the video clip.

[0182] After calculating the similarity between each image frame and its adjacent image frames in a video segment, the execution device can further calculate the average similarity of multiple image frames in the same video segment. Specifically, the execution device can calculate the average similarity d between multiple image frames. i The average similarity of image frames in the video segment is obtained by calculating the average value.

[0183] Step 3: Calculate the elimination ratio of video segments.

[0184] Since the memory size of the accelerator running the AI ​​model is fixed, the features generated by the accelerator when processing the AI ​​model cannot exceed the size of the accelerator memory. At this time, the execution device can determine the elimination ratio of each video segment based on the average similarity corresponding to each video segment under the constraint of the accelerator memory, as shown in Formula 3 below.

[0185] Where, α i C represents the percentage of video segment i to be removed. max L represents the maximum number of feature segments supported by the accelerator memory, T represents the total number of image frames included in the target video, and N represents the number of image blocks divided for each image frame during processing.

[0186] Phase four involves processing video segments using an AI model and uniformly removing all feature segments corresponding to the same video segment based on the removal ratio of the video segments and the attention scores of the generated feature segments.

[0187] Specifically, the execution device processes the video features of each video segment sequentially using an AI model to generate a set of feature segments corresponding to each video segment. Furthermore, after generating a set of feature segments for a video segment, the execution device removes some feature segments based on the video segment elimination ratio and the attention score of each feature segment in the set.

[0188] For example, the execution device first calculates the attention score of the feature segments generated by each neural network layer in the AI ​​model for the video segment based on the features of the prompt words. It then sorts all feature segments corresponding to the video segment in descending order of attention score. Based on the elimination ratio corresponding to the video segment, it determines which feature segments with the lowest attention scores to eliminate. After determining the feature segments to be eliminated, the execution device can find the required feature segments in each layer of feature segments generated by each neural network layer in the AI ​​model for the video segment, and eliminate these found feature segments, thus obtaining the retained feature segments in each layer.

[0189] In this way, for the same video segment generated by different neural network layers, the same elimination ratio is no longer assigned to each neural network layer. Instead, the feature segments of all neural network layers are uniformly selected for elimination according to their attention scores, and different elimination ratios are assigned to different neural network layers to achieve inter-layer adaptive allocation of elimination ratios.

[0190] It should be noted that, for video segments other than the first video segment, when the execution device processes these video segments using the AI ​​model, it also needs to consider the retained feature segments corresponding to the already processed video segments.

[0191] The method provided in this application has been described in detail above. Next, the device provided in this application for performing the above method will be described.

[0192] Please refer to Figure 8, which is a schematic diagram of the structure of a video processing apparatus provided in this application. As shown in Figure 8, the video processing apparatus includes: an acquisition module 801, used to acquire multiple video segments of a target video, wherein each video segment includes one or more image frames; a processing module 802, used to determine the similarity of a first video segment among the multiple video segments, wherein the similarity of the first video segment is used to indicate the degree of similarity between image frames in the first video segment and adjacent image frames; the processing module 802 is further used to remove some feature segments from a first set of feature segments based on the similarity of the first video segment, to obtain retained feature segments, wherein the retained feature segments include the feature segments in the first set of feature segments that were not removed, the first set of feature segments is obtained by processing the first video segment using an artificial intelligence (AI) model, and the retained feature segments are used to input into the AI ​​model to perform video processing tasks.

[0193] In one possible implementation, the processing module 802 is further configured to: remove some feature segments from the first group of feature segments based on a first removal ratio, wherein the first removal ratio is used to indicate the ratio between the amount of data of the feature segments to be removed from the first group of feature segments and the total amount of data of the first group of feature segments; wherein the first removal ratio is related to the similarity of the first video segment.

[0194] In one possible implementation, the processing module 802 is further configured to: based on a second elimination ratio of the second video segment among multiple video segments, eliminate some feature segments from the second group of feature segments corresponding to the second video segment; wherein the similarity of the first video segment is different from that of the second video segment, and the first elimination ratio and the second elimination ratio are different, and the retained feature segments also include the feature segments in the second group of feature segments that were not eliminated.

[0195] In one possible implementation, the processing module 802 is further configured to: determine a first elimination ratio based on the similarity between the first video segment and the similarity between other video segments among the multiple video segments.

[0196] In one possible implementation, the processing module 802 is further configured to: determine a first elimination ratio based on the similarity ratio between the similarity of the first video segment and the total similarity; wherein the total similarity is obtained based on the similarity of the multiple video segments respectively.

[0197] In one possible implementation, the similarity of the first video segment is used to indicate one or more of the following: the degree of similarity between pixels of an image frame in the first video segment and pixels of adjacent image frames; or, the degree of similarity between vectors corresponding to image frames in the first video segment and vectors corresponding to adjacent image frames; or, the degree of similarity between feature segments corresponding to image frames in the first video segment and feature segments corresponding to adjacent image frames.

[0198] In one possible implementation, the first set of feature segments includes multi-layer feature segments, one layer of which is extracted by a neural network layer in the AI ​​model; the processing module 802 is further configured to: remove some feature segments from the multi-layer feature segments, wherein the amount of data of the feature segments removed from each layer of the multi-layer feature segments is related to the importance of each layer of feature segments.

[0199] In one possible implementation, the AI ​​model is a video large language model. The acquisition module 801 is also used to acquire the prompt words corresponding to the target video. The prompt words are used to indicate the video processing task. The processing module 802 is also used to calculate the attention score between each feature segment and the feature of the prompt word in the multi-layer feature segments. The attention score of a feature segment is used to indicate the importance of the feature segment.

[0200] In one possible implementation, the features of the prompt word include multiple feature segments; the processing module 802 is further configured to obtain the attention score between the features of the first feature segment and the prompt word based on the sum of the attention scores between the first feature segment in the multi-layer feature segments and each feature segment corresponding to the prompt word.

[0201] In one possible implementation, multiple video segments include a second video segment, and the first video segment is located before the second video segment in the target video; the processing module 802 is further configured to process the second video segment using an AI model based on the feature segments retained in the first set of feature segments to obtain the second set of feature segments corresponding to the second video segment.

[0202] Both the acquisition module 801 and the processing module 802 can be implemented in software or in hardware. For example, the implementation of the processing module 802 will be described below. Similarly, the implementation of the acquisition module 801 can be referenced from the implementation of the processing module 802.

[0203] As an example of a software functional unit, processing module 802 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, processing module 802 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0204] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0205] As an example of a hardware functional unit, the processing module 802 may include at least one computing device, such as a server. Alternatively, the processing module 802 may be implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system-on-chip (SoC), an offload card, an accelerator card, or any combination thereof.

[0206] The processing module 802 includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the processing module 802 can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the processing module 802 can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offloading cards, and accelerator cards.

[0207] Please refer to Figure 9, which is a schematic diagram of the structure of a computing device provided in this application. The computing device 900 shown in Figure 9 can be used to execute the video processing method provided in this embodiment. As shown in Figure 9, the computing device 900 includes: a bus 902, a processor 904, a memory 906, and a communication interface 908. The processor 904, the memory 906, and the communication interface 908 communicate with each other via the bus 902. The computing device 900 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 900.

[0208] Bus 902 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 9, but this does not imply that there is only one bus or one type of bus. Bus 902 can include pathways for transmitting information between various components of computing device 900 (e.g., memory 906, processor 904, communication interface 908).

[0209] Processor 904 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0210] The memory 906 may include volatile memory, such as random access memory (RAM). The processor 904 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0211] The memory 906 stores executable program code, which the processor 904 executes to implement the functions of the aforementioned acquisition module and processing module, thereby realizing the video processing method of the model described above. In other words, the memory 906 stores instructions for executing the video processing method of the model.

[0212] The communication interface 908 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 900 and other devices or communication networks.

[0213] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0214] Please refer to Figure 10, which is a schematic diagram of a computing device cluster provided in this application. As shown in Figure 10, the computing device cluster includes at least one computing device 900. The memory 906 of one or more computing devices 900 in the computing device cluster may store the same instructions for executing video processing methods.

[0215] In some possible implementations, the memory 906 of one or more computing devices 900 in the computing device cluster may also store partial instructions for executing video processing methods. In other words, a combination of one or more computing devices 900 can jointly execute the instructions for executing video processing methods.

[0216] It should be noted that the memory 906 in different computing devices 900 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the video processing device. That is, the instructions stored in the memory 906 of different computing devices 900 can implement the functions of one or more of the aforementioned acquisition and processing modules.

[0217] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 11 illustrates one possible implementation. Figure 11 is also a schematic diagram of another computing device cluster structure provided in this application. As shown in Figure 11, in computing device cluster 1100, two computing devices 900A and 900B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 906 in computing device 900A stores instructions for executing the functions of the acquisition module. Simultaneously, the memory 906 in computing device 900B stores instructions for executing the functions of the processing module.

[0218] It should be understood that the functions of computing device 900A shown in Figure 11 can also be performed by multiple computing devices 900. Similarly, the functions of computing device 900B can also be performed by multiple computing devices 900.

[0219] This application also provides a chip comprising a processing unit and a communication unit. The processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in a storage unit to cause the chip within the electronic device to perform the methods described in the above embodiments. Optionally, the storage unit may be an in-chip storage unit, such as a register or cache. Alternatively, the storage unit may be an external storage unit located within a wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0220] Referring to Figure 12, which is a schematic diagram of the structure of a computer-readable storage medium provided in this application. This application also provides a computer-readable storage medium in which, in some embodiments, the method disclosed in Figure 2A above can be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of art.

[0221] Figure 12 schematically illustrates a conceptual partial view of an example computer-readable storage medium arranged according to at least some of the embodiments shown herein, the example computer-readable storage medium including a computer program for executing computer processes on a computing device.

[0222] In one embodiment, the computer-readable storage medium 1200 is provided using a signal bearer medium 1201. The signal bearer medium 1201 may include one or more program instructions 1202 that, when executed by one or more processors, can provide the functions or parts thereof described above with reference to FIG. 2A.

[0223] In some examples, signal carrying medium 1201 may include computer-readable medium 1203, such as, but not limited to, hard disk drive, compact disc (CD), digital video disc (DVD), digital magnetic tape, memory, ROM or RAM, etc.

[0224] In some embodiments, the signal-bearing medium 1201 may comprise a computer-recordable medium 1204, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, and so on. In some embodiments, the signal-bearing medium 1201 may comprise a communication medium 1205, such as, but not limited to, digital and / or analog communication media (e.g., fiber optic cables, waveguides, wired communication links, wireless communication links, and so on). Therefore, for example, the signal-bearing medium 1201 may be transmitted by a wireless form of communication medium 1205 (e.g., a wireless communication medium conforming to the IEEE 1202.X standard or other transmission protocols).

[0225] One or more program instructions 1202 may be, for example, computer-executable instructions or logical implementation instructions. In some examples, the computing device may be configured to provide various operations, functions, or actions in response to one or more program instructions 1202 conveyed to the computing device via a computer-readable medium 1203, a computer-recordable medium 1204, and / or a communication medium 1205.

[0226] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0227] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods of the various embodiments of this application.

[0228] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0229] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, computer instructions can be transferred from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A video processing method, characterized in that, include: Acquire multiple video segments of the target video, wherein each video segment includes one or more image frames; Determine the similarity of a first video segment among the plurality of video segments, wherein the similarity of the first video segment is used to indicate the degree of similarity between image frames in the first video segment and adjacent image frames; Based on the similarity of the first video segment, some feature segments are removed from the first set of feature segments to obtain the retained feature segments. The retained feature segments include the feature segments that were not removed from the first set of feature segments. The first set of feature segments is obtained by processing the first video segment through an artificial intelligence (AI) model. The retained feature segments are used as inputs into the AI ​​model to perform video processing tasks.

2. The method according to claim 1, characterized in that, The step of removing some feature segments from the first group of feature segments based on the similarity of the first video segment includes: Based on a first elimination ratio, some feature segments are eliminated from the first group of feature segments. The first elimination ratio is used to indicate the ratio between the amount of data of the feature segments to be eliminated in the first group of feature segments and the total amount of data of the first group of feature segments. The first elimination ratio is related to the similarity of the first video segment.

3. The method according to claim 2, characterized in that, The method further includes: Based on a second elimination ratio for the second video segment among multiple video segments, some feature segments are eliminated from the second group of feature segments corresponding to the second video segment; wherein, the similarity between the first video segment and the second video segment is different, and the first elimination ratio and the second elimination ratio are different, and the retained feature segments also include feature segments in the second group of feature segments that were not eliminated.

4. The method according to claim 2 or 3, characterized in that, The method further includes: The first elimination ratio is determined based on the similarity between the first video segment and the similarity between other video segments among the plurality of video segments.

5. The method according to any one of claims 2-4, characterized in that, Determining the first rejection ratio includes: The first elimination ratio is determined based on the similarity ratio between the first video segment and the total similarity; wherein the total similarity is obtained based on the similarity of each of the multiple video segments.

6. The method according to any one of claims 1-4, characterized in that, The similarity of the first video segment is used to indicate one or more of the following: The degree of similarity between the pixels of an image frame in the first video clip and the pixels of adjacent image frames; Alternatively, the degree of similarity between the vectors corresponding to the image frames in the first video segment and the vectors corresponding to the adjacent image frames; Alternatively, the degree of similarity between the feature segments corresponding to the image frames in the first video segment and the feature segments corresponding to the adjacent image frames.

7. The method according to any one of claims 1-6, characterized in that, The first set of feature segments includes multi-layer feature segments, one of which is extracted by a neural network layer in the AI ​​model; The step of removing some feature segments from the first set of feature segments includes: In the multi-layer feature segments, some feature segments are removed, and the amount of data of the feature segments removed from each layer of the multi-layer feature segments is related to the importance of each layer of feature segments.

8. The method according to claim 7, characterized in that, The AI ​​model is a video-based large language model, and the method further includes: Obtain the prompt word corresponding to the target video; the prompt word is used to instruct the video processing task. Calculate the attention score between each feature segment in the multi-layer feature segments and the feature of the prompt word. The attention score of a feature segment is used to indicate the importance of the feature segment.

9. The method according to claim 8, characterized in that, The features of the prompt words include multiple feature segments; The calculation of the attention score between each feature segment in the multi-layer feature segment and the feature of the prompt word includes: The attention score between the first feature segment and the feature of the prompt word is obtained by summing the attention scores between the first feature segment and each feature segment corresponding to the prompt word in the multi-layer feature segment.

10. The method according to any one of claims 1-9, characterized in that, The plurality of video segments includes a second video segment, and the first video segment is located before the second video segment in the target video, the method comprising: Based on the feature segments retained in the first set of feature segments, the second video segment is processed by the AI ​​model to obtain the second set of feature segments corresponding to the second video segment.

11. A video processing apparatus, characterized in that, include: The acquisition module is used to acquire multiple video segments of the target video, wherein the video segments include one or more image frames; The processing module is used to determine the similarity of a first video segment among the plurality of video segments, wherein the similarity of the first video segment is used to indicate the degree of similarity between image frames in the first video segment and adjacent image frames; The processing module is further configured to remove some feature segments from the first set of feature segments based on the similarity of the first video segment, thereby obtaining retained feature segments. The retained feature segments include the feature segments that were not removed from the first set of feature segments. The first set of feature segments is obtained by processing the first video segment using an artificial intelligence (AI) model. The retained feature segments are used to input into the AI ​​model to perform video processing tasks.

12. The apparatus according to claim 11, characterized in that, The processing module is further configured to: Based on a first elimination ratio, some feature segments are eliminated from the first group of feature segments. The first elimination ratio is used to indicate the ratio between the amount of data of the feature segments to be eliminated in the first group of feature segments and the total amount of data of the first group of feature segments. The first elimination ratio is related to the similarity of the first video segment.

13. The apparatus according to claim 12, characterized in that, The processing module is further configured to: Based on a second elimination ratio for the second video segment among multiple video segments, some feature segments are eliminated from the second group of feature segments corresponding to the second video segment; wherein, the similarity between the first video segment and the second video segment is different, and the first elimination ratio and the second elimination ratio are different, and the retained feature segments also include feature segments in the second group of feature segments that were not eliminated.

14. The apparatus according to claim 12 or 13, characterized in that, The processing module is further configured to: The first elimination ratio is determined based on the similarity between the first video segment and the similarity between other video segments among the plurality of video segments.

15. The apparatus according to any one of claims 12-14, characterized in that, The processing module is further configured to: The first elimination ratio is determined based on the similarity ratio between the first video segment and the total similarity; wherein the total similarity is obtained based on the similarity of each of the multiple video segments.

16. The apparatus according to any one of claims 11-14, characterized in that, The similarity of the first video segment is used to indicate one or more of the following: The degree of similarity between the pixels of an image frame in the first video clip and the pixels of adjacent image frames; Alternatively, the degree of similarity between the vectors corresponding to the image frames in the first video segment and the vectors corresponding to the adjacent image frames; Alternatively, the degree of similarity between the feature segments corresponding to the image frames in the first video segment and the feature segments corresponding to the adjacent image frames.

17. The apparatus according to any one of claims 11-16, characterized in that, The first set of feature segments includes multi-layer feature segments, one of which is extracted by a neural network layer in the AI ​​model; The processing module is further configured to: In the multi-layer feature segments, some feature segments are removed, and the amount of data of the feature segments removed from each layer of the multi-layer feature segments is related to the importance of each layer of feature segments.

18. The apparatus according to claim 17, characterized in that, The AI ​​model is a video-based large language model. The acquisition module is further configured to acquire prompt words corresponding to the target video, the prompt words being used to instruct the video processing task; The processing module is also used to calculate the attention score between each feature segment in the multi-layer feature segments and the feature of the prompt word, and the attention score of a feature segment is used to indicate the importance of the feature segment.

19. The apparatus according to claim 18, characterized in that, The features of the prompt words include multiple feature segments; The processing module is further configured to obtain the attention score between the first feature segment and the feature of the prompt word based on the sum of the attention scores between the first feature segment in the multi-layer feature segments and each feature segment corresponding to the prompt word.

20. The apparatus according to any one of claims 11-19, characterized in that, The plurality of video segments includes a second video segment, and the first video segment is located before the second video segment in the target video; The processing module is further configured to process the second video segment using the AI ​​model based on the feature segments retained in the first set of feature segments, to obtain the second set of feature segments corresponding to the second video segment.

21. A computing device, characterized in that, The device includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the computing device performs the method as described in any one of claims 1 to 10.

22. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the operational steps of the method as described in any one of claims 1 to 10.

23. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed by the computer, cause the computer to perform the method according to any one of claims 1 to 10.

24. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1 to 10.

Images