Method and apparatus for video bullet-comment moderation, electronic device, and computer storage medium
By extracting frames from the video and using a multi-feature neural network dual-tower model and a large-scale review model to automatically review the bullet screen content, the problem of low efficiency and high cost in existing video bullet screen review technologies has been solved. This has enabled fast, efficient and accurate bullet screen content review, reducing the review burden and improving efficiency.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- HUNAN HAPPLY SUNSHINE INTERACTIVE ENTERTAINMENT MEDIA CO LTD
- Filing Date
- 2025-04-09
- Publication Date
- 2026-05-15
AI Technical Summary
Existing video comment moderation systems are inefficient, costly, and difficult to accurately review comments based on the video context. Traditional manual moderation is inefficient, sensitive word filtering methods have low recall rates, and BERT inference is slow, failing to meet real-time requirements.
By extracting frames from the video, the semantic review model and the large review model are used to automatically review the bullet screen content and video information of each frame. The multi-layer neural network model of the semantic review model and the large review model is combined to review the bullet screen content. The weights of the pre-trained model are optimized by using a multi-feature neural network dual-tower model and a low-rank matrix to achieve fast and efficient bullet screen content review.
It has achieved fast, efficient and accurate review of bullet screen content, reduced the review burden, saved costs, improved review efficiency, and improved accuracy and timeliness through inspection and review.
Smart Images
Figure CN2025087985_15052026_PF_FP_ABST
Abstract
Description
Methods, devices, electronic equipment, and computer storage media for reviewing video bullet comments
[0001] This invention claims priority to Chinese Patent Application No. 202411586850.4, filed with the Chinese Patent Office on November 8, 2024, entitled "Method, Apparatus, Electronic Device and Computer Storage Medium for Reviewing Video Bullet Comments", the entire contents of which are incorporated herein by reference. Technical Field
[0002] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device, and computer storage medium for reviewing video bullet comments. Background Technology
[0003] In the internet industry, video comment content moderation is an indispensable part. Traditional moderation systems mainly rely on human reviewers, which, while accurate, is inefficient and costly. Another approach combines machine-assisted review with human review, with representative technologies including sensitive word filtering and BERT, enabling automated text review. Sensitive word filtering methods offer high accuracy and speed, but suffer from low recall and difficulty in comprehensive coverage. BERT, a pre-trained language model, can also be used for text classification, but its accuracy is lower than human review, and its inference speed is very slow, making it unsuitable for real-time review scenarios. Furthermore, these methods can only review the comment content itself, failing to consider the video context. In fact, the same comment may receive different review results on different videos. Summary of the Invention
[0004] In view of this, this application provides a method, apparatus, electronic device, and computer storage medium for reviewing video bullet comments, which can quickly, efficiently, and accurately review bullet comment content automatically, greatly reducing the review burden, saving review costs, and improving review efficiency.
[0005] The first aspect of this application provides a method for reviewing video bullet comments, including:
[0006] Frames are extracted from the target video to obtain the bullet comments and video information for each frame;
[0007] For each frame's bullet comment content and video information, the bullet comment content and video information of the frame are input into a semantic review model to obtain the review score of the bullet comment content of the frame; wherein, the semantic review model is obtained by training a text classification model based on a first training sample set; the first training sample set includes at least one first training sample; the first training sample includes bullet comment content and video text information corresponding to a first historical frame;
[0008] The review result of the bullet screen content of the frame is determined based on the review score of the bullet screen content of the frame; wherein, if the review result of the bullet screen content of the frame is passed, the bullet screen content of the frame is exposed; if the review result of the bullet screen content of the frame is failed, the bullet screen content of the frame is blocked.
[0009] Optionally, if the review result of the bullet screen content of the frame is approved, then after revealing the bullet screen content of the frame, the method further includes:
[0010] The displayed bullet comments will be treated as pending bullet comments.
[0011] At preset time intervals, for each pending bullet comment content, the pending bullet comment content and its corresponding video information are input into the large-scale review model, and the inspection and review results of the pending bullet comment content are output. The large-scale review model is obtained by fine-tuning a pre-trained model based on a second training sample set. The second training sample set includes at least one second training sample. The second training sample includes training task instructions, training sample questions, and training sample answers. The training sample questions include the bullet comment content and video information corresponding to historical frames.
[0012] If the inspection and review results of the pending bullet screen content are passed, the pending bullet screen content will be marked and stored in the database;
[0013] If the inspection and review results of the pending bullet screen content are not satisfactory, the pending bullet screen content will be recalled.
[0014] Optionally, the method for constructing the audit model includes:
[0015] Construct a second training sample set; wherein the second training sample set includes at least one second training sample; the second training sample includes training task instructions, training sample questions, and training sample answers; the training sample questions include the bullet screen content and video text information corresponding to the second historical frame;
[0016] Fix the pre-training weights of the pre-trained model;
[0017] All the second training samples are fed into the first low-rank matrix and the second low-rank matrix respectively for training, to obtain the trained first low-rank matrix and the trained second low-rank matrix;
[0018] The target weights are determined based on the trained first low-rank matrix, the trained second low-rank matrix, and the pre-trained weights.
[0019] Update the target weights to the audit model.
[0020] Optionally, determining the target weights based on the trained first low-rank matrix, the trained second low-rank matrix, and the pre-trained weights includes:
[0021] The trained first low-rank matrix and the trained second low-rank matrix are multiplied by matrix multiplication to obtain the weights of the trained low-rank matrix.
[0022] The target weights are obtained by adding the trained low-rank matrix weights to the pre-trained weights.
[0023] Optionally, the method for constructing the semantic review model includes:
[0024] Construct a first training sample set; wherein the first training sample set includes at least one first training sample; the first training sample includes the bullet screen content and video text information corresponding to the first historical frame;
[0025] For each first training sample, the first training sample is input into the large-scale audit model, and the label of the first training sample is output.
[0026] The bullet screen content and video text content corresponding to the first historical frame are vectorized to obtain the first vector and the second vector, respectively.
[0027] Generate a fusion vector for the first training sample based on the first vector and the second vector;
[0028] The text classification model is trained based on the fusion vectors of all the first training samples and the labels of the first training samples to obtain the semantic review model.
[0029] Optionally, generating the fusion vector of the first training sample based on the first vector and the second vector includes:
[0030] The first vector is input into the first gated loop unit and outputs the first target vector; the first vector is input into the first highway unit and outputs the second target vector.
[0031] The second vector is input into the second gated loop unit and the third target vector is output; the second vector is input into the second highway unit and the fourth target vector is output.
[0032] Soft attention processing is performed based on the second target vector and the fourth target vector to obtain the first attention vector and the second attention vector;
[0033] The first target vector, the second target vector, and the first attention vector are concatenated to obtain the first concatenated vector;
[0034] The third target vector, the fourth target vector, and the second attention vector are concatenated to obtain the second concatenated vector.
[0035] The first concatenated vector is input into the first gated loop unit to obtain the fifth target vector, and the second concatenated vector is input into the second gated loop unit to obtain the sixth target vector;
[0036] The fusion vector of the first training sample is obtained by performing function operations based on the fifth target vector and the sixth target vector.
[0037] Optionally, the step of performing a function operation based on the fifth target vector and the sixth target vector to obtain the fusion vector of the first training sample includes:
[0038] The difference between the fifth target vector and the sixth target vector is taken as the seventh target vector;
[0039] The product of the fifth target vector and the sixth target vector is taken as the eighth target vector;
[0040] The fifth target vector, the seventh target vector, and the eighth target vector are concatenated to obtain the fusion vector of the first training sample.
[0041] A second aspect of this application provides a device for reviewing video bullet comments, comprising:
[0042] The frame extraction unit is used to extract frames from the target video to obtain the bullet screen content and video information for each frame.
[0043] The first input unit is used to input the bullet comment content and video information of each frame into the semantic review model to obtain the review score of the bullet comment content of the frame; wherein, the semantic review model is obtained by training a text classification model based on a first training sample set; the first training sample set includes at least one first training sample; the first training sample includes the bullet comment content and video text information corresponding to the first historical frame;
[0044] The first determining unit is used to determine the review result of the bullet screen content of the frame based on the review score of the bullet screen content of the frame; wherein, if the review result of the bullet screen content of the frame is passed, the bullet screen content of the frame is exposed; if the review result of the bullet screen content of the frame is failed, the bullet screen content of the frame is blocked.
[0045] Optionally, the video bullet comment review device further includes:
[0046] The second determining unit is used to take the exposed bullet screen content as the bullet screen content to be determined;
[0047] The second input unit is used to input the pending bullet comment content and the corresponding video information into the large-scale review model at preset time intervals for each pending bullet comment content, and output the inspection and review result of the pending bullet comment content; wherein, the large-scale review model is obtained by fine-tuning a pre-trained model based on a second training sample set; the second training sample set includes at least one second training sample; the second training sample includes training task instructions, training sample questions, and training sample answers; the training sample questions include the bullet comment content and video information corresponding to historical frames;
[0048] The storage unit is used to mark and store the pending bullet screen content in the database if the inspection and review results of the pending bullet screen content pass.
[0049] The recall unit is used to recall the pending bullet screen content if the inspection and review results show that the pending bullet screen content fails.
[0050] Optionally, the building blocks of the audit large model include:
[0051] The first construction unit is used to construct a second training sample set; wherein the second training sample set includes at least one second training sample; the second training sample includes training task instructions, training sample questions, and training sample answers; the training sample questions include the bullet screen content and video text information corresponding to the second historical frame;
[0052] Fixed units are used to fix the pre-training weights of the pre-trained model;
[0053] The feeding unit is used to feed all the second training samples into the first low-rank matrix and the second low-rank matrix respectively for training, so as to obtain the trained first low-rank matrix and the trained second low-rank matrix.
[0054] The third determining unit is used to determine the target weights based on the trained first low-rank matrix, the trained second low-rank matrix, and the pre-trained weights.
[0055] An update unit is used to update the target weights to the audit big model.
[0056] Optionally, the third determining unit includes:
[0057] The first calculation unit is used to calculate the weights of the trained low-rank matrix by performing matrix multiplication on the trained first low-rank matrix and the trained second low-rank matrix.
[0058] The second calculation unit is used to add the trained low-rank matrix weights to the pre-trained weights to obtain the target weights.
[0059] Optionally, the building blocks of the semantic review model include:
[0060] The second construction unit is used to construct a first training sample set; wherein the first training sample set includes at least one first training sample; the first training sample includes the bullet screen content and video text information corresponding to the first historical frame;
[0061] The third input unit is used to input the first training sample into the audit model for each first training sample and output the label of the first training sample.
[0062] The vectorization unit uses language to vectorize the bullet screen content and video text content corresponding to the first historical frame to obtain the first vector and the second vector.
[0063] A fusion unit is configured to generate a fusion vector of the first training sample based on the first vector and the second vector;
[0064] The training unit is used to train the text classification model based on the fusion vector of all the first training samples and the label of the first training samples to obtain the semantic review model.
[0065] Optionally, the fusion unit includes:
[0066] The fourth input unit is used to input the first vector into the first gated loop unit and output the first target vector, and to input the first vector into the first highway unit and output the second target vector.
[0067] The fifth input unit is used to input the second vector into the second gated loop unit and output the third target vector, and to input the second vector into the second highway unit and output the fourth target vector.
[0068] A soft attention processing unit is configured to perform soft attention processing based on the second target vector and the fourth target vector to obtain a first attention vector and a second attention vector.
[0069] The first splicing unit is used to splice the first target vector, the second target vector, and the first attention vector to obtain a first spliced vector;
[0070] The second splicing unit is used to splice the third target vector, the fourth target vector, and the second attention vector to obtain a second spliced vector;
[0071] The sixth input unit is used to input the first concatenated vector into the first gated loop unit to obtain the fifth target vector, and to input the second concatenated vector into the second gated loop unit to obtain the sixth target vector;
[0072] The function operation unit is used to perform function operations based on the fifth target vector and the sixth target vector to obtain the fusion vector of the first training sample.
[0073] Optionally, the function operation unit includes:
[0074] The third calculation unit is used to take the difference between the fifth target vector and the sixth target vector as the seventh target vector;
[0075] The fourth calculation unit is used to take the product of the fifth target vector and the sixth target vector as the eighth target vector;
[0076] The third splicing unit is used to splice the fifth target vector, the seventh target vector, and the eighth target vector to obtain the fusion vector of the first training sample.
[0077] A third aspect of this application provides an electronic device, comprising:
[0078] One or more processors;
[0079] A storage device on which one or more programs are stored;
[0080] When the one or more programs are executed by the one or more processors, the one or more processors implement the video bullet screen review method as described in any one of the first aspects.
[0081] The fourth aspect of this application provides a computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the video bullet screen review method as described in any one of the first aspects.
[0082] As can be seen from the above scheme, this application provides a method, apparatus, electronic device, and computer storage medium for reviewing video bullet comments. By extracting frames from the target video to obtain the bullet comment content and video information for each frame, the bullet comment content and video information of each frame are input into a semantic review model to obtain a review score for the bullet comment content of the frame. Based on the review score, the review result of the bullet comment content of the frame is determined. If the review result of the bullet comment content of the frame is passed, the bullet comment content of the frame is displayed; if the review result of the bullet comment content of the frame is failed, the bullet comment content of the frame is blocked. This achieves the goal of quickly, efficiently, and accurately automatically reviewing bullet comment content, greatly reducing the review burden, saving review costs, and improving review efficiency. Attached Figure Description
[0083] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0084] Figure 1 is a flowchart of a video bullet comment review method provided in an embodiment of this application;
[0085] Figure 2 is a flowchart of a method for constructing a semantic auditing model according to another embodiment of this application;
[0086] Figure 3 is a flowchart of a method for constructing a large audit model according to another embodiment of this application;
[0087] Figure 4 is a flowchart of a method for generating a fusion vector according to another embodiment of this application;
[0088] Figure 5 is a schematic diagram of a gated loop unit provided in another embodiment of this application;
[0089] Figure 6 is a schematic diagram of the structure of a semantic auditing model provided in another embodiment of this application;
[0090] Figure 7 is a flowchart of a video bullet screen review method provided in another embodiment of this application;
[0091] Figure 8 is a schematic diagram of a video bullet screen review device provided in another embodiment of this application;
[0092] Figure 9 is a schematic diagram of an electronic device for implementing a method for reviewing video bullet comments according to another embodiment of this application. Detailed Implementation
[0093] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0094] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.
[0095] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.
[0096] It should be noted that the concepts of "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.
[0097] It should be noted that the terms "a" and "a plurality of" used in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".
[0098] This application provides a method for reviewing video bullet comments, as shown in Figure 1, which specifically includes the following steps:
[0099] S101. Extract frames from the target video to obtain the bullet comments and video information for each frame.
[0100] The video information includes, but is not limited to, video dialogue and video description, etc., which are not limited here.
[0101] It is understandable that videos are usually stored as video data, so it is necessary to extract frames from the video to obtain the bullet comments and video information for each frame. The implementation of frame extraction can be, but is not limited to, using the Python programming language and the OpenCV library for video frame extraction processing, and no restrictions are imposed here.
[0102] S102. For each frame's bullet comment content and video information, input the bullet comment content and video information of that frame into the semantic review model to obtain the review score of the bullet comment content of that frame.
[0103] The semantic review model is obtained by training the text classification model based on the first training sample set; the first training sample set includes at least one first training sample; the first training sample includes the bullet screen content and video text information corresponding to the first historical frame.
[0104] In the specific implementation of this application, the semantic auditing model can be called Bpool, which is a dual-tower neural network model based on multiple features. In another embodiment of this application, one implementation method of constructing the semantic auditing model is shown in Figure 2, including:
[0105] S201. Construct the first training sample set.
[0106] The first training sample set includes at least one first training sample; the first training sample includes the bullet screen content and video text information corresponding to the first historical frame.
[0107] S202. For each first training sample, input the first training sample into the large audit model and output the label of the first training sample.
[0108] The review model is obtained by fine-tuning the pre-trained model based on the second training sample set; the second training sample set includes at least one second training sample; the second training sample includes training task instructions, training sample questions, and training sample answers; the training sample questions include the bullet screen content and video information corresponding to the historical frames.
[0109] In the specific implementation of this application, the pre-trained model can be, but is not limited to, GLM-9b, which is a transformer-based pre-trained large language model, and is not limited here.
[0110] Specifically, by using a large-scale data verification model to label the first training samples, the labels for the first training samples are obtained, eliminating the need for manual data labeling as in existing technologies, effectively reducing labor costs. Furthermore, the large-scale data verification model requires less training data than the semantic data verification model, and its accuracy is higher. Therefore, the large-scale data verification model can be trained with a small amount of data first, and then used to generate training data for the semantic data verification model, significantly reducing manual labeling costs.
[0111] In the specific implementation of this application, the first training sample may be preprocessed in a format that the large review model can recognize before inputting the first training sample into the review model, or the first training sample may be preprocessed by the large review model after inputting the first training sample into the review model to obtain a format that the large review model can recognize. No limitation is made here.
[0112] Optionally, in another embodiment of this application, one implementation of the method for constructing a large audit model, as shown in Figure 3, includes:
[0113] S301. Construct the second training sample set.
[0114] The second training sample set includes at least one second training sample; the second training sample includes training task instructions, training sample questions, and training sample answers; the training sample questions include the bullet screen content and video text information corresponding to the second historical frame.
[0115] In the specific implementation process of this application, the sample data will first undergo preprocessing steps such as regularization and data cleaning to obtain a second training sample that meets the format requirements, which is not limited here.
[0116] Specifically, the training task instruction is the task instruction given to the large-scale model for review, which allows the large-scale model to understand what the task is. For example: Based on the content of the bullet comments and the video information, please determine whether the bullet comments belong to a preset category: normal, abusive, dangerous, etc. There are no restrictions here.
[0117] The second training sample that meets the format requirements can be, but is not limited to, the format of training task instruction-training sample question-training sample answer.
[0118] Here is an example of the second training sample:
[0119] “instruct”: “Based on the content of the bullet comments and the video information, please determine whether the bullet comments belong to the preset categories: abusive language and dangerous language. If they belong to these categories, please output ‘fail’; otherwise, output ‘pass’.”
[0120] “text”:“Haha, it’s really beautiful”;
[0121] “video_info”: “Video Description: This video tells the story of Snow White. Video Type: Animation. Video Dialogue: XXXXXXXX”;
[0122] “label”: “Pass”.
[0123] Here, 'instruct' represents the training task instruction, 'text' represents the bullet screen content, 'video_info' represents the video information, and 'label' represents the training sample response.
[0124] S302, Fix the pre-training weights of the pre-trained model.
[0125] Among them, the pre-training weights of the pre-trained model are fixed, that is, only the pre-training weights of the pre-trained model are loaded during training, and the pre-training weights of the pre-trained model are not updated.
[0126] In the specific implementation of this application, low-rank adaptive fine-tuning techniques (LORA techniques) can be used to fix the pre-training weights of the pre-trained model, and no limitation is made here.
[0127] S303. Feed all the second training samples into the first low-rank matrix and the second low-rank matrix respectively for training to obtain the trained first low-rank matrix and the trained second low-rank matrix.
[0128] First, the second training sample is embedded, i.e., vectorized, turning a single text data into a vector. The training target is the label of this data (the review result 0 and 1). These vectors are combined to form the final weight C. The training process involves updating the parameters of this weight C. The weight C is a matrix that we want to fine-tune. Assuming it is m*n dimensional, LoRa fine-tuning involves performing a low-rank decomposition on this m*n matrix, splitting it into two low-rank matrices, m*d and d*n, namely the first low-rank matrix A and the second low-rank matrix B, thereby reducing the amount of computation and improving training efficiency.
[0129] S304. Determine the target weights based on the trained first low-rank matrix, the trained second low-rank matrix, and the pre-trained weights.
[0130] Optionally, in another embodiment of this application, one implementation of step S304 includes:
[0131] First, the trained first low-rank matrix and the trained second low-rank matrix are multiplied by matrix to obtain the weights of the trained low-rank matrix; then, the weights of the trained low-rank matrix are added to the pre-trained weights to obtain the target weights.
[0132] The pre-trained weights are matrices. The weights of the trained low-rank matrix (m*n) are calculated by multiplying the trained first low-rank matrix and the trained second low-rank matrix, where m*n = m*d + d*n. The weights of the trained low-rank matrix are then added to the pre-trained weights to obtain the target weights.
[0133] S305. Update the target weights to the audit model.
[0134] S203. Vectorize the bullet screen content and video text content corresponding to the first historical frame to obtain the first vector and the second vector.
[0135] The first vector is the vector obtained by vectorizing the bullet screen content corresponding to the first historical frame; the second vector is the vector obtained by vectorizing the video text content corresponding to the first historical frame.
[0136] In the specific implementation of this application, the first vector and the second vector can be, but are not limited to, vectors with n*30*128 dimensions.
[0137] S204. Generate the fusion vector of the first training sample based on the first vector and the second vector.
[0138] Optionally, in another embodiment of this application, one implementation of step S204, as shown in FIG4, includes:
[0139] S401. Input the first vector into the first gated loop unit and output the first target vector; input the first vector into the first highway unit and output the second target vector.
[0140] It should be noted that the gated recurrent unit (GRU) consists of two gates, as shown in Figure 5: an update gate and a reset gate. When the msg vector (the first vector) comes in, the initial state is h(t-1). First, the gate values z(t) and r(t) of the update gate and the reset gate are calculated. Then, these two gate values are calculated according to a preset calculation method. Finally, the two are added together to obtain the final output h(t), which is the first target vector.
[0141] Among them, the highway unit is a network structure that performs a nonlinear transformation on a part of the input (such as a part of the first vector), and directly skips another part of the input (such as another part of the first vector) without any change. Then the two are merged to obtain the first target vector.
[0142] S402, input the second vector into the second gated loop unit and output the third target vector, and input the second vector into the second highway unit and output the fourth target vector.
[0143] It should be noted that the specific implementation method of step S402 can be found in step S401, and will not be repeated here.
[0144] S403. Perform soft attention processing based on the second target vector and the fourth target vector to obtain the first attention vector and the second attention vector.
[0145] Specifically, firstly, the dot product of the second and fourth target vectors is calculated to obtain the attention points of the second and fourth target vectors. Then, the attention points are activated twice using the second and fourth target vectors respectively to obtain the activated second and fourth target vectors. Next, the dot product of the activated second and fourth target vectors is calculated to obtain the first attention vector. Finally, the dot product of the activated fourth target vectors is calculated to obtain the second attention vector.
[0146] S404. Concatenate the first target vector, the second target vector, and the first attention vector to obtain the first concatenated vector.
[0147] Specifically, the first target vector, the second target vector, and the first attention vector are concatenated (using the concat operation to concatenate vectors) to obtain the first concatenated vector.
[0148] S405. Concatenate the third target vector, the fourth target vector, and the second attention vector to obtain the second concatenated vector.
[0149] It should be noted that the specific implementation method of step S405 can be found in step S404, and will not be repeated here.
[0150] S406. Input the first concatenation vector into the first gated loop unit to obtain the fifth target vector, and input the second concatenation vector into the second gated loop unit to obtain the sixth target vector.
[0151] Specifically, the processing method of the first gated loop unit for the first concatenated vector and the processing method of the second gated loop unit for the second concatenated vector can be found in step S401. Both are processing of vectors by the gated loop unit. However, step S401 processes the first vector (which can also be understood as the initial vector), while step S406 processes the first concatenated vector obtained after various processing of the first vector. These details will not be elaborated here.
[0152] S407. Perform function operations based on the fifth target vector and the sixth target vector to obtain the fusion vector of the first training sample.
[0153] Optionally, in another embodiment of this application, one implementation of step S407 includes:
[0154] First, the difference between the fifth and sixth target vectors is used as the seventh target vector. Then, the product of the fifth and sixth target vectors is used as the eighth target vector. Finally, the fifth, seventh, and eighth target vectors are concatenated to obtain the fusion vector of the first training sample.
[0155] S205. Train the text classification model based on the fusion vectors of all first training samples and the labels of the first training samples to obtain the semantic review model.
[0156] Specifically, training involves updating the parameters of the weights. The training target is the target label (0-1). The parameters of the target weights are continuously updated based on the training target. The loss function measures the difference between the predicted and the true values; the smaller the loss function, the better the training. Training stops when the loss function drops to a reasonable level.
[0157] Figure 6 shows the structural diagram of the semantic moderation model (Bpool). `msg` represents the bullet screen content, and `audio_info` represents the video text information. First, `msg` and `audio_info` are converted into a format that the model can process, and then vectorized to obtain a first vector and a second vector. The first and second vectors are then fed into two towers in Bpool, named the `msg` tower and the `audio_info` tower, respectively. Only the processing of the `msg` tower is described here; the processing of the `audio_info` tower is the same as that of the `msg` tower. The `msg` vector (first vector) is fed into the GRU (Gated Recurrent Unit) and the highway (Highway Unit). After passing through the GRU, `a_encoder` (first target vector) is obtained; after passing through the highway, `a_highway` (second target vector) is obtained. Then, soft attention is applied to `a_highway` (second target vector) and `b_highway` (fourth target vector) to obtain `a_attention` (first attention vector) and `b_attention` (second attention vector). Next, a_encoder, a_highway, and a_attention are concatenated to obtain a_merge (the first concatenated vector). a_merge is then fed into aGRU to obtain apool (the fifth target vector). Similarly, bpool (the sixth target vector) can be obtained. Then, apool and bpool are processed using functions (diff, mul, etc.), and concatenated with apool to obtain a fused vector. Finally, the text classification model is trained based on all the fused vectors and the labels output by the main model for semantic review, resulting in a semantic review model.
[0158] S103. Determine the review result of the bullet screen content of the frame based on the review score of the bullet screen content of the frame.
[0159] If the review result of the bullet screen content of a frame is approved, the bullet screen content of that frame will be displayed; if the review result of the bullet screen content of a frame is not approved, the bullet screen content of that frame will be blocked.
[0160] In the specific implementation of this application, it can be pre-set that if the review score is greater than 0.5, the review will not be approved, and if it is less than 0.5, the review will be approved. No restrictions are imposed here.
[0161] Optionally, in another embodiment of this application, if the review result of the bullet screen content of the frame is passed, then after the bullet screen content of the frame is exposed, one implementation of the video bullet screen review method, as shown in Figure 7, further includes:
[0162] S701. The exposed bullet screen content will be used as the bullet screen content to be determined.
[0163] S702. At preset intervals, for each pending bullet comment, the pending bullet comment content and the corresponding video information are input into the review model, and the inspection and review results of the pending bullet comment content are output.
[0164] The review model is obtained by fine-tuning the pre-trained model based on the second training sample set; the second training sample set includes at least one second training sample; the second training sample includes training task instructions, training sample questions, and training sample answers; the training sample questions include the bullet screen content and video information corresponding to the historical frames.
[0165] S703. If the inspection and review results of the pending bullet screen content are passed, the pending bullet screen content will be marked and stored in the database.
[0166] In the specific implementation of this application, the pending bullet screen content is marked and stored in the database. If bullet screen content that is consistent with the pending bullet screen content is encountered again in the future, there is no need to review it again, and the bullet screen content can be directly exposed.
[0167] Understandably, a time limit could be set, where if a comment identical to the pending comment is encountered within that time limit, it would be displayed directly without further review. After that time limit, the pending comment would be removed from the database.
[0168] S704. If the inspection and review results of the pending bullet screen content are not satisfactory, the pending bullet screen content will be recalled.
[0169] It should be noted that in the specific implementation of this application, when using the large-scale review model for inspection and review, it is an offline inspection. The bullet comments that fail the review are then recalled and intercepted, thereby maximizing the accuracy and timeliness of video bullet comment review.
[0170] As can be seen from the above scheme, this application provides a method for reviewing video bullet comments. By extracting frames from the target video to obtain the bullet comment content and video information for each frame, the bullet comment content and video information of each frame are input into a semantic review model to obtain a review score for the bullet comment content of that frame. Based on the review score, the review result of the bullet comment content for that frame is determined. Specifically, if the review result of the bullet comment content for that frame is passed, the bullet comment content of that frame is displayed; if the review result of the bullet comment content for that frame is failed, the bullet comment content of that frame is blocked. This achieves the goal of quickly, efficiently, and accurately automatically reviewing bullet comment content, greatly reducing the review burden, saving review costs, and improving review efficiency.
[0171] This application provides a video bullet comment review device, as shown in Figure 8, specifically including:
[0172] The frame extraction unit 801 is used to extract frames from the target video to obtain the bullet screen content and video information for each frame.
[0173] The first input unit 802 is used to input the bullet screen content and video information of each frame into the semantic review model to obtain the review score of the bullet screen content of that frame.
[0174] The semantic review model is obtained by training the text classification model based on the first training sample set; the first training sample set includes at least one first training sample; the first training sample includes the bullet screen content and video text information corresponding to the first historical frame.
[0175] The first determining unit 803 is used to determine the review result of the bullet screen content of the frame based on the review score of the bullet screen content of the frame.
[0176] If the review result of the bullet screen content of a frame is approved, the bullet screen content of that frame will be displayed; if the review result of the bullet screen content of a frame is not approved, the bullet screen content of that frame will be blocked.
[0177] The specific working process of the unit disclosed in the above embodiments of this application can be found in the corresponding method embodiments, as shown in Figure 1, and will not be repeated here.
[0178] Optionally, in another embodiment of this application, one implementation of the video bullet comment review device further includes:
[0179] The second determining unit is used to take the exposed bullet screen content as the bullet screen content to be determined.
[0180] The second input unit is used to input the pending bullet comment content and the corresponding video information into the review model at preset intervals, and output the inspection and review results of the pending bullet comment content.
[0181] The review model is obtained by fine-tuning the pre-trained model based on the second training sample set; the second training sample set includes at least one second training sample; the second training sample includes training task instructions, training sample questions, and training sample answers; the training sample questions include the bullet screen content and video information corresponding to the historical frames.
[0182] The storage unit is used to mark and store the pending bullet screen content in the database if the inspection and review results show that the pending bullet screen content has passed.
[0183] The recall unit is used to recall the pending bullet screen content if the inspection and review results show that the pending bullet screen content fails.
[0184] The specific working process of the unit disclosed in the above embodiments of this application can be found in the corresponding method embodiments, as shown in Figure 7, and will not be repeated here.
[0185] Optionally, in another embodiment of this application, one implementation of reviewing the building blocks of a large model includes:
[0186] The first building unit is used to build the second training sample set.
[0187] The second training sample set includes at least one second training sample; the second training sample includes training task instructions, training sample questions, and training sample answers; the training sample questions include the bullet screen content and video text information corresponding to the second historical frame.
[0188] Fixed units are used to fix the pre-training weights of a pre-trained model.
[0189] The feeding unit is used to feed all the second training samples into the first low-rank matrix and the second low-rank matrix respectively for training, so as to obtain the trained first low-rank matrix and the trained second low-rank matrix.
[0190] The third determining unit is used to determine the target weights based on the trained first low-rank matrix, the trained second low-rank matrix, and the pre-trained weights.
[0191] The update unit is used to update the target weights to the audit model.
[0192] The specific working process of the unit disclosed in the above embodiments of this application can be found in the corresponding method embodiments, as shown in Figure 3, and will not be repeated here.
[0193] Optionally, in another embodiment of this application, one implementation of the third determining unit includes:
[0194] The first calculation unit is used to perform matrix multiplication on the trained first low-rank matrix and the trained second low-rank matrix to obtain the weights of the trained low-rank matrix.
[0195] The second calculation unit is used to add the trained low-rank matrix weights to the pre-trained weights to obtain the target weights.
[0196] For details on the specific working process of the units disclosed in the above embodiments of this application, please refer to the corresponding method embodiments, which will not be repeated here.
[0197] Optionally, in another embodiment of this application, one implementation of the semantic review model construction unit includes:
[0198] The second building unit is used to build the first training sample set.
[0199] The first training sample set includes at least one first training sample; the first training sample includes the bullet screen content and video text information corresponding to the first historical frame.
[0200] The third input unit is used to input the first training sample into the large model for each first training sample and output the label of the first training sample.
[0201] The vectorization unit uses language to vectorize the bullet screen content and video text content corresponding to the first historical frame, respectively, to obtain the first vector and the second vector.
[0202] A fusion unit is used to generate a fusion vector for the first training sample based on the first vector and the second vector.
[0203] The training unit is used to train the text classification model based on the fusion vector of all first training samples and the label of the first training samples to obtain the semantic review model.
[0204] The specific working process of the unit disclosed in the above embodiments of this application can be found in the corresponding method embodiments, as shown in Figure 2, and will not be repeated here.
[0205] Optionally, in another embodiment of this application, one implementation of the fusion unit includes:
[0206] The fourth input unit is used to input the first vector into the first gated loop unit and output the first target vector, and to input the first vector into the first highway unit and output the second target vector.
[0207] The fifth input unit is used to input the second vector into the second gated loop unit and output the third target vector, and to input the second vector into the second highway unit and output the fourth target vector.
[0208] The soft attention processing unit is used to perform soft attention processing based on the second target vector and the fourth target vector to obtain the first attention vector and the second attention vector.
[0209] The first concatenation unit is used to concatenate the first target vector, the second target vector, and the first attention vector to obtain the first concatenation vector.
[0210] The second concatenation unit is used to concatenate the third target vector, the fourth target vector, and the second attention vector to obtain the second concatenation vector.
[0211] The sixth input unit is used to input the first concatenated vector into the first gated loop unit to obtain the fifth target vector, and to input the second concatenated vector into the second gated loop unit to obtain the sixth target vector.
[0212] The function operation unit is used to perform function operations based on the fifth target vector and the sixth target vector to obtain the fusion vector of the first training sample.
[0213] The specific working process of the unit disclosed in the above embodiments of this application can be found in the corresponding method embodiments, as shown in Figure 4, and will not be repeated here.
[0214] Optionally, in another embodiment of this application, one implementation of the function operation unit includes:
[0215] The third calculation unit is used to take the difference between the fifth target vector and the sixth target vector as the seventh target vector.
[0216] The fourth calculation unit is used to take the product of the fifth target vector and the sixth target vector as the eighth target vector.
[0217] The third concatenation unit is used to concatenate the fifth target vector, the seventh target vector, and the eighth target vector to obtain the fusion vector of the first training sample.
[0218] For details on the specific working process of the units disclosed in the above embodiments of this application, please refer to the corresponding method embodiments, which will not be repeated here.
[0219] As can be seen from the above scheme, this application provides a video bullet comment review device. The device extracts frames from the target video using a frame extraction unit 801 to obtain the bullet comment content and video information for each frame. For each frame, the first input unit 802 inputs the bullet comment content and video information into a semantic review model to obtain a review score for the bullet comment content. The first determining unit 803 determines the review result of the bullet comment content based on the review score. If the review result of the bullet comment content for a frame is satisfactory, the bullet comment content for that frame is displayed; if the review result of the bullet comment content for that frame is unsatisfactory, the bullet comment content for that frame is blocked. This achieves the goal of quickly, efficiently, and accurately automatically reviewing bullet comment content, greatly reducing the review burden, saving review costs, and improving review efficiency.
[0220] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.
[0221] Another embodiment of this application provides an electronic device, as shown in FIG9, including:
[0222] One or more processors 901.
[0223] Storage device 902, on which one or more programs are stored.
[0224] When the one or more programs are executed by the one or more processors 901, the one or more processors 901 implement the video bullet screen review method as described in any of the above embodiments.
[0225] Another embodiment of this application provides a computer storage medium storing a computer program thereon, wherein when the computer program is executed by a processor, it implements the video bullet screen review method as described in any of the above embodiments.
[0226] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0227] It should be noted that the computer-readable medium described above in this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.
[0228] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.
[0229] Another embodiment of this application provides a computer program product, which, when executed, is used to perform the above-described method for reviewing video bullet comments.
[0230] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, it performs the functions defined in the methods of the embodiments of this application.
[0231] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in this application is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely exemplary forms of implementing this application.
[0232] While several specific implementation details are included in the foregoing discussion, these should not be construed as limiting the scope of this application. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.
[0233] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described application concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions claimed in this application.
Claims
1. A method for reviewing video bullet comments, characterized in that, include: Frames are extracted from the target video to obtain the bullet comments and video information for each frame; For each frame's bullet comment content and video information, the bullet comment content and video information of the frame are input into a semantic review model to obtain the review score of the bullet comment content of the frame; wherein, the semantic review model is obtained by training a text classification model based on a first training sample set; the first training sample set includes at least one first training sample; the first training sample includes bullet comment content and video text information corresponding to a first historical frame; The review result of the bullet screen content of the frame is determined based on the review score of the bullet screen content of the frame; wherein, if the review result of the bullet screen content of the frame is passed, the bullet screen content of the frame is exposed; if the review result of the bullet screen content of the frame is failed, the bullet screen content of the frame is blocked.
2. The method for reviewing video bullet comments according to claim 1, characterized in that, If the review result of the bullet screen content of the frame is approved, then after revealing the bullet screen content of the frame, the process further includes: The displayed bullet comments will be treated as pending bullet comments. At preset time intervals, for each pending bullet comment, the pending bullet comment and its corresponding video information are input into the large-scale review model, and the inspection and review results of the pending bullet comment are output. The large-scale review model is obtained by fine-tuning a pre-trained model based on a second training sample set. The second training sample set includes at least one second training sample. The second training sample includes training task instructions, training sample questions, and training sample answers. The training sample questions include the bullet comment content and video information corresponding to historical frames. If the inspection and review results of the pending bullet screen content are passed, the pending bullet screen content will be marked and stored in the database; If the inspection and review results of the pending bullet screen content are not satisfactory, the pending bullet screen content will be recalled.
3. The method for reviewing video bullet comments according to claim 2, characterized in that, The method for constructing the large-scale audit model includes: Construct a second training sample set; wherein the second training sample set includes at least one second training sample; the second training sample includes training task instructions, training sample questions, and training sample answers; the training sample questions include the bullet screen content and video text information corresponding to the second historical frame; Fix the pre-training weights of the pre-trained model; All the second training samples are fed into the first low-rank matrix and the second low-rank matrix respectively for training, to obtain the trained first low-rank matrix and the trained second low-rank matrix; The target weights are determined based on the trained first low-rank matrix, the trained second low-rank matrix, and the pre-trained weights. Update the target weights to the audit model.
4. The method for reviewing video bullet comments according to claim 3, characterized in that, The step of determining the target weights based on the trained first low-rank matrix, the trained second low-rank matrix, and the pre-trained weights includes: The trained first low-rank matrix and the trained second low-rank matrix are multiplied by matrix multiplication to obtain the weights of the trained low-rank matrix. The target weights are obtained by adding the trained low-rank matrix weights to the pre-trained weights.
5. The method for reviewing video bullet comments according to claim 3, characterized in that, The method for constructing the semantic review model includes: Construct a first training sample set; wherein the first training sample set includes at least one first training sample; the first training sample includes the bullet screen content and video text information corresponding to the first historical frame; For each first training sample, the first training sample is input into the large-scale audit model, and the label of the first training sample is output. The bullet screen content and video text content corresponding to the first historical frame are vectorized to obtain the first vector and the second vector, respectively. Generate a fusion vector for the first training sample based on the first vector and the second vector; The text classification model is trained based on the fusion vectors of all the first training samples and the labels of the first training samples to obtain the semantic review model.
6. The method for reviewing video bullet comments according to claim 5, characterized in that, The step of generating the fusion vector of the first training sample based on the first vector and the second vector includes: The first vector is input into the first gated loop unit and outputs the first target vector; the first vector is input into the first highway unit and outputs the second target vector. The second vector is input into the second gated loop unit and the third target vector is output; the second vector is input into the second highway unit and the fourth target vector is output. Soft attention processing is performed based on the second target vector and the fourth target vector to obtain the first attention vector and the second attention vector; The first target vector, the second target vector, and the first attention vector are concatenated to obtain the first concatenated vector; The third target vector, the fourth target vector, and the second attention vector are concatenated to obtain the second concatenated vector. The first concatenated vector is input into the first gated loop unit to obtain the fifth target vector, and the second concatenated vector is input into the second gated loop unit to obtain the sixth target vector; The fusion vector of the first training sample is obtained by performing function operations based on the fifth target vector and the sixth target vector.
7. The method for reviewing video bullet comments according to claim 6, characterized in that, The step of performing function operations based on the fifth target vector and the sixth target vector to obtain the fusion vector of the first training sample includes: The difference between the fifth target vector and the sixth target vector is taken as the seventh target vector; The product of the fifth target vector and the sixth target vector is taken as the eighth target vector; The fifth target vector, the seventh target vector, and the eighth target vector are concatenated to obtain the fusion vector of the first training sample.
8. A video bullet comment review device, characterized in that, include: The frame extraction unit is used to extract frames from the target video to obtain the bullet screen content and video information for each frame. The first input unit is used to input the bullet comment content and video information of each frame into the semantic review model to obtain the review score of the bullet comment content of the frame; wherein, the semantic review model is obtained by training a text classification model based on a first training sample set; the first training sample set includes at least one first training sample; the first training sample includes the bullet comment content and video text information corresponding to the first historical frame; The first determining unit is used to determine the review result of the bullet screen content of the frame based on the review score of the bullet screen content of the frame; wherein, if the review result of the bullet screen content of the frame is passed, the bullet screen content of the frame is exposed; if the review result of the bullet screen content of the frame is failed, the bullet screen content of the frame is blocked.
9. An electronic device, characterized in that, include: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the video bullet screen review method as described in any one of claims 1 to 7.
10. A computer storage medium, characterized in that, It stores a computer program, wherein the computer program, when executed by a processor, implements the video bullet screen review method as described in any one of claims 1 to 7.