Fake review detection method based on BERT-cnn

Abstract

The present invention provides a fake review detection method based on a BERT-CNN, comprising: first using a BERT model to capture a deep representation from text data, wherein the deep structure of a plurality of Encoder layers in a BERT enables the BERT to capture language information of different levels ranging from a shallow grammar feature to a deep semantic feature; then inputting the deep representation into a convolutional neural network (CNN), so as to enable a sequence to more completely learn context information; and combining the information learned by the BERT and the information of the CNN, performing processing by means of a fully connected layer, and using a softmax classifier to implement classification. In the present invention, the BERT model is used, and the BERT model abandons a traditional neural network structure, and is combined with the characteristics of a Transformer model, thereby obtaining a more excellent result while reducing the amount of calculation and improving the parallel efficiency, solving the problem of inaccurate detection of current fake reviews, and improving the accuracy and robustness of detection.

Description

A method for detecting fake reviews based on BERT-CNN Technical Field

[0001] This invention relates to a method for detecting fake comments based on BERT-CNN, and belongs to the fields of natural language processing and deep learning. Background Technology

[0002] With the rapid development of B2C e-commerce, consumers rely heavily on online reviews before selecting and purchasing products online, making product reviews quite important.

[0003] Fake reviews cause significant distress to consumers, preventing them from obtaining accurate information about a product and hindering their ability to purchase what they truly want. Furthermore, after-sales service is time-consuming and inconvenient, leading to unnecessary losses. For businesses, fake reviews have both advantages and disadvantages. On one hand, paid reviews can increase product awareness, but malicious negative reviews from competitors can also cause considerable trouble. On the other hand, genuine reviews encourage product upgrades and iterations, leading to technological advancements that benefit the industry. Conversely, industry upgrades can lower product prices, benefiting consumers. Therefore, fake reviews have a serious impact on both businesses and consumers, making research on fake reviews of significant social and economic value.

[0004] Current research on fake reviews mainly includes traditional fake review detection methods, graph structure-based methods, and machine learning-based methods. Traditional fake review research is based on behavioral and linguistic features, and this type of research is relatively early. Graph structure-based detection methods connect reviews, reviewers, and products into a graph structure, treating the detection problem as a classification, ranking, or joint classification problem. Machine learning for fake review detection requires manually labeling training sets and selecting features. However, it struggles with high-dimensional feature extraction, resulting in poor model training. Essentially, this is because the learning ability of neural networks is not yet sufficient, the models are not yet perfect, and computational efficiency needs improvement.

[0005] Summary of the Invention

[0006] The purpose of this invention is to solve the problems of low recognition rate and poor robustness of fake comments on the Internet in the prior art, and to provide a fake comment detection method based on BERT-CNN.

[0007] Technical solution

[0008] This invention uses the BERT model to learn from comment content, extract features, fully learn sentence information, and mine the information within it. Then, it uses a CNN for context learning to improve the accuracy of detecting fake comments. The specific solution is as follows:

[0009] A method for detecting fake comments based on BERT-CNN includes the following steps:

[0010] S1. Obtain the comment text dataset and preprocess the obtained dataset;

[0011] S2. Input the processed dataset into the BERT model. The BERT model consists of multiple encoder layers, including a positional encoding layer, a multi-head attention layer, an addition and normalization layer, and a feedforward network layer. After passing through multiple encoder layers, the BERT masking language task and the next sentence prediction task are completed, and the text word vectors trained by BERT are obtained.

[0012] S3. Input the word vectors of the text trained by BERT into the convolutional neural network to perform local semantic learning of the text;

[0013] S4. Obtain the label results by classifying using a softmax classifier.

[0014] Furthermore, in step S1, the preprocessing includes removing unnecessary symbols, processing the text format, and uniformly replacing symbols for numbers and characters.

[0015] Furthermore, in step S2, the masking language task refers to randomly masking a portion of words in a sentence, and then simultaneously using contextual information to predict the masked words, so as to better understand the meaning of words based on the whole text.

[0016] The next sentence prediction task refers to enabling the model to better understand the relationships between sentences.

[0017] Furthermore, in step S2, the positional encoding layer employs different strategies to implement different positional encodings. After positional encoding, an encoding matrix X with the same dimension as the input is obtained. pos When it is superimposed on the original word embeddings, a new word embedding matrix is ​​obtained, as shown in equation (1) X embedding =X embedding +X pos (1).

[0018] Furthermore, in step S2, the multi-head attention layer consists of multiple scaling points and a self-attention mechanism;

[0019] Attention mechanisms map queries and a set of key-value pairs to a high-dimensional space, that is, by performing a weighted summation of the values, where the weights are calculated from the query and the key.

[0020] For scaled dot product attention, the query vector (Q) and the key vector (K) are multiplied by a vector, and then processed by a vector multiplication operation of size 1. Scaling, d k This represents the vector dimension of the query vector (Q) and the key vector (K); after normalization, it is multiplied by the value vector (V) to obtain the final output, as shown in equation (2):

[0021] Q, K, and V employ a self-attention mechanism, starting from their own input X. embedding This is generated by performing a linear mapping, as shown in equations (3), (4), and (5): Q = X embedding *W Q (3) K = X embedding *W K (4) V=X embedding *W V (5)

[0022] All three weight matrices are square matrices with dimension Embedding;

[0023] Multi-head attention concatenates the outputs of multiple scaled dot product attention methods, and it has multiple sets of W. Q W K W V The weight matrix, each set of weight matrices is randomly initialized, X embedding The vector is divided into multiple parts, multiplied by the weights, and the resulting vector forms the input vector W. i X, as shown in equations (6), (7), and (8), generates Q. i K i V i Let i = 1, ..., h, where h represents the number of heads to be segmented. Calculate the Attention weight matrix to obtain each head, then merge the heads and pass them through a fully connected layer (W). O The transformation yields the final result, as shown in equations (9) and (10): Q i =QW i Q ,i=1,…,h(6) K i =KW i K ,i=1,…,h(7) V i =VW i Vi = 1, ..., h(8) head i =Attention(Q) i K i V i ), i=1,…,h(9) MultiHead(Q,K,V)=Concat(head1,…,head n W O (10).

[0024] Furthermore, in step S2, the addition and normalization layer includes an Add&Norm layer. Add adds the input and output of the self-attention layer (residual connection), and LayerNorm performs layer normalization, normalizing the hidden layers in the neural network to a standard normal distribution, thereby accelerating convergence.

[0025] Furthermore, in step S2, the feedforward network layer consists of two fully connected layers plus a ReLU activation function, as shown in equation (11): FFN(x)=max(0,xW1+b1)W2+b2 (11)

[0026] Where W1 and b1 represent the weights and biases of the first fully connected layer, respectively, and W2 and b2 represent the weights and biases of the second fully connected layer, respectively.

[0027] Furthermore, in step S3, the convolutional neural network includes an embedding layer, a convolutional layer, a pooling layer, and a fully connected layer;

[0028] In the embedding layer, the text word vectors trained by BERT are combined into an n×m matrix Z = [W1,…,W1]. n ], where W i The word vectors output by BERT;

[0029] Convolutional layers extract convolutional features from the input text. Each convolutional kernel is like a window. By continuously moving the window, semantic learning is performed on the input text to extract different semantic features.

[0030] The convolution kernel is convolved with the input matrix, and the formula for convolution is shown in equation (12): S=f(WZ+b)(12)

[0031] Where S represents the feature matrix after the convolution operation, W represents the weight matrix, and b represents the offset vector;

[0032] After convolution, feature mapping is required. Each convolutional layer contains multiple feature maps. The mapping operation is performed by the ReLU function, as shown in equation (13): ReLU=max(0,x)(13)

[0033] The pooling layer serves to select and filter features, simplifying computational complexity and extracting key features.

[0034] The role of the fully connected layer is to combine the extracted features, use the learned features to achieve the learning objective, and send the output value to the classifier.

[0035] Furthermore, in step S4, the softmax classifier uses the softmax function to calculate the probability. To prevent data overflow, the largest value among the output values ​​is subtracted from each output value, as shown in equations (14) and (15): D = max(Z) (14)

[0036] Where Z = [X] [CLS] ,Z1,…,Z n ], X [CLS] The output after training with BERT is [Z1,…,Z]. n [Z1, ..., Zn] represents the feature matrix extracted by CNN after BERT training, where D is [Z1, ..., Zn]. n The maximum value in ];

[0037] Finally, the cross-entropy function is used to continuously update and optimize the parameters of the model, as shown in equation (16):

[0038] Where C represents the size of the training set, y c p represents the tag value of the comment. c The model classification probability for the comments.

[0039] The beneficial effects of this invention are:

[0040] This invention provides a method for detecting fake reviews based on BERT-CNN. First, the review data is preprocessed. Then, BERT is used to extract global semantic features of the text. BERT consists of multiple encoder layers and can learn the semantic features of the context better than general neural network models. Then, the learned vector matrix is ​​input into the CNN for local semantic extraction. Finally, the output layer combines and outputs the results. Compared with existing technologies, the method of this invention improves the accuracy of detection and solves the problem of poor accuracy of fake reviews. Attached Figure Description

[0041] Figure 1 is a flowchart of the fake comment detection method based on BERT-CNN in Example 1;

[0042] Figure 2 is a diagram of the overall model of BERT in Example 1;

[0043] Figure 3 is a diagram of the Encoder layer model of BERT in Example 1;

[0044] Figure 4 is a model design diagram of the CNN in Example 1. Detailed Implementation

[0045] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings.

[0046] Example 1

[0047] As shown in Figure 1, a fake comment detection method based on BERT-CNN includes the following steps:

[0048] S1. First, obtain the comment text dataset through web crawling. Preprocess the obtained dataset so that the processed data can be better used by the BERT model for feature learning, thereby improving the training completion rate.

[0049] The preprocessing specifically includes: removing unnecessary symbols, processing the text format, and uniformly replacing symbols for numbers and characters;

[0050] S2. Input the data into the BERT model. The overall model diagram of BERT is shown in Figure 2. Through training the BERT model, complete the BERT masking language task and the next sentence prediction task (the masking language task is to randomly mask a part of the words in the sentence and then use the context information to predict the masked words in order to better understand the meaning of the words based on the whole text; the next sentence prediction task is to enable the model to better understand the relationship between sentences), and obtain the text word vectors after being trained by BERT;

[0051] The basic structure of BERT consists of multiple encoder layers, as shown in Figure 3. The encoder layers include a position encoding layer, a multi-head attention layer, an addition and normalization layer, and a feedforward network layer.

[0052] After inputting the data into the word embedding layer of the BERT model, each character in the sentence is transformed into a dimensional vector. Therefore, Input: X = [X1, X2, ..., X...] n X becomes X after embedding. embedding The dimensions include the number of sentences in a batch, the number of characters in each sentence, and the embedding dimension of each character.

[0053] Next, the input enters the positional encoding layer, where the position is marked. After positional encoding, an encoding matrix X with the same dimension as the input is obtained. pos When he superimposed it onto the original word embeddings, a new word embedding matrix is ​​obtained, as shown in equation (1): X embedding =X embedding +X pos (1)

[0054] Next, the position-encoded word embedding matrix is ​​input into a multi-head attention layer, which consists of multiple scaling points and a self-attention mechanism;

[0055] Attention mechanisms map queries and a set of key-value pairs to a high-dimensional space, which involves weighted summation of the values, where the weights are calculated from the query and the key.

[0056] For scaled dot product attention, the query vector (Q) and the key vector (K) are multiplied by a vector, and then processed by a vector multiplication operation of size 1. Scaling, d k This represents the vector dimension of the query vector (Q) and the key vector (K). After normalization, it is multiplied by the value vector (V) to obtain the final output, as shown in equation (2):

[0057] Q, K, and V employ a self-attention mechanism, starting from their own input X. embedding This is generated by performing a linear mapping, as shown in equations (3), (4), and (5): Q = X embedding *W Q (3) K = X embedding *W K (4) V=X embedding *W V (5)

[0058] All three weight matrices are square matrices with dimension Embedding;

[0059] Multi-head attention concatenates the outputs of multiple scaled dot product attention methods, and it has multiple sets of W. Q W K W V The weight matrix, each set of weight matrices is randomly initialized, X embedding The vector is divided into multiple parts, multiplied by the weights, and the resulting vector forms the input vector W. i X, as shown in equations (6), (7), and (8), generates Q. i K i V i Let i = 1, ..., h, where h represents the number of heads to be segmented. Calculate the Attention weight matrix to obtain each head, then merge the heads and pass them through a fully connected layer (W). O The transformation yields the final result, as shown in equations (9) and (10): Q i =QW i Q ,i=1,...,h(6) K i =KW iK ,i=1,...,h(7) V i =VW i V i = 1, ..., h(8) head i =Attention(Q) i K i V i ), i=1,…,h(9) MultiHead(Q,K,V)=Concat(head1,…,head n W O (10).

[0060] Next, the vector output from the multi-head attention layer is input into the addition and normalization layer, which includes the Add & Norm layer. Add adds the input and output of the self-attention layer (residual connection), and LayerNorm performs layer normalization, normalizing the hidden layers in the neural network to a standard normal distribution, thus accelerating convergence.

[0061] Next, we move on to the feedforward network layer (FFN), which consists of two fully connected layers plus a ReLU activation function, as shown in equation (11): FFN(x)=max(0,xW1+b1)W2+b2(11)

[0062] Here, W1 and b1 represent the weights and biases of the first fully connected layer, respectively, and W2 and b2 represent the weights and biases of the second fully connected layer, respectively. Each fully connected layer is normalized using LayerNorm.

[0063] After the FFN layer, an Add&Norm layer is performed, as described above, to obtain a brand new word output vector, which is then input into the next Encoder layer; after multiple Encoder layers, the two tasks of BERT are achieved.

[0064] After passing through the FFN layer, an Add&Norm layer is performed, as described above, to obtain a brand new word output vector, which is then input into the next Encoder layer. After passing through multiple Encoder layers, the two tasks of BERT are achieved.

[0065] S3. Input the word vectors of the text trained by BERT into a convolutional neural network (CNN) to perform local semantic learning of the text;

[0066] The design of the convolutional neural network is shown in Figure 4, including an embedding layer, a convolutional layer, a pooling layer, and a fully connected layer.

[0067] In the embedding layer, the text word vectors trained by BERT are combined into an n×m matrix Z = [W1,…,W1].n ], where W i The word vectors output by BERT;

[0068] Convolutional layers extract convolutional features from the input text. Each convolutional kernel is like a window. By continuously moving the window, semantic learning is performed on the input text to extract different semantic features.

[0069] The convolution kernel is convolved with the input matrix, and the formula for convolution is shown in equation (12): S=f(WZ+b)(12)

[0070] Where S represents the feature matrix after the convolution operation, W represents the weight matrix, and b represents the offset vector;

[0071] After convolution, feature mapping is required. Each convolutional layer contains multiple feature maps. Each feature map is a "plane" composed of multiple neurons. A convolutional filter is used to extract a feature of the input. The mapping operation is completed by the ReLU function, as shown in equation (13): ReLU=max(0,x)(13)

[0072] Next, we enter the pooling layer, which is responsible for feature selection and filtering, simplifying computational complexity and extracting key features.

[0073] Finally, we arrive at the output layer, which includes a fully connected layer and a softmax output. The role of the fully connected layer is to combine the extracted features, use the learned features to achieve the learning objective, and send the output value to the classifier.

[0074] S4. Obtain the label results by classifying using a softmax classifier.

[0075] The classifier uses the softmax function to calculate probabilities. To prevent data overflow, the largest output value is subtracted from each output value, as shown in equations (14) and (15): D = max(Z) (14)

[0076] Where Z = [X] [CLS] ,Z1,…,Z n ], X [CLS] The output after training with BERT is [Z1,…,Z]. n [Z1, ..., Zn] represents the feature matrix extracted by CNN after BERT training, where D is [Z1, ..., Zn]. n The maximum value in ];

[0077] Finally, the cross-entropy function is used to continuously update and optimize the parameters of the model, as shown in equation (16):

[0078] Where C represents the size of the training set, y c p represents the tag value of the comment. c The model classification probability for the comments.

[0079] The above description is only a preferred embodiment of the present invention. For those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for detecting fake reviews based on BERT-CNN, characterized in that, Includes the following steps: S1. Obtain the comment text dataset and preprocess the obtained dataset; S2. Input the processed dataset into the BERT model. The BERT model consists of multiple encoder layers, including a positional encoding layer, a multi-head attention layer, an addition and normalization layer, and a feedforward network layer. After passing through multiple encoder layers, the BERT masking language task and the next sentence prediction task are completed, and the text word vectors trained by BERT are obtained. S3. Input the word vectors of the text trained by BERT into the convolutional neural network to perform local semantic learning of the text; S4. Obtain the label results by classifying using a softmax classifier.

2. The method for detecting fake reviews based on BERT-CNN according to claim 1, characterized in that, In step S1, the preprocessing includes removing unnecessary symbols, processing the text format, and uniformly replacing symbols for numbers and characters.

3. The method for detecting fake reviews based on BERT-CNN according to claim 1, characterized in that, In step S2, the masking language task refers to randomly masking a portion of words in a sentence and then using contextual information to predict the masked words in order to better understand the meaning of the words based on the whole text. The next sentence prediction task refers to enabling the model to better understand the relationships between sentences.

4. The method for detecting fake reviews based on BERT-CNN according to claim 1, characterized in that, In step S2, the positional encoding layer employs different strategies to implement different positional encodings. After positional encoding, an encoding matrix X with the same dimension as the input is obtained. pos When he superimposed it onto the original word embeddings, a new word embedding matrix is ​​obtained, as shown in equation (1): X embedding =X embedding +X pos (1).

5. The method for detecting fake reviews based on BERT-CNN according to claim 1, characterized in that, In step S2, the multi-head attention layer consists of multiple scaling points and a self-attention mechanism; Attention mechanisms map queries and a set of key-value pairs to a high-dimensional space, that is, by performing a weighted summation of the values, where the weights are calculated from the query and the key. For scaled dot product attention, the query vector (Q) and the key vector (K) are multiplied by a vector, and then processed by a vector multiplication operation of size 1. Scaling, d k This represents the vector dimension of the query vector (Q) and the key vector (K); after normalization, it is multiplied by the value vector (V) to obtain the final output, as shown in equation (2): Q, K, and V employ a self-attention mechanism, starting from their own input X. embedding This is generated by performing a linear mapping, as shown in equations (3), (4), and (5): Q=X embedding *W Q (3) K=X embedding *W K (4) V=X embedding *W V (5) All three weight matrices are square matrices with dimension Embedding; Multi-head attention concatenates the outputs of multiple scaled dot product attention methods, and it has multiple sets of W. Q W K W V The weight matrix, each set of weight matrices is randomly initialized, X embedding The vector is divided into multiple parts, multiplied by the weights, and the resulting vector forms the input vector W. i X, as shown in equations (6), (7), and (8), generates Q. i K i V i Let i = 1, ..., h, where h represents the number of heads to be segmented. Calculate the Attention weight matrix to obtain each head, then merge the heads and pass them through a fully connected layer (W). O The transformation yields the final result, as shown in equations (9) and (10): Q i =QW i Q ,i=1,…,h(6) K i =KW i K ,i=1,…,h(7) V i =VW i V ,i=1,…,h(8) head i =Attention(Q i ,K i ,V i ),i=1,…,h(9) MultiHead(Q,K,V)=Concat(head1,…,head n )W O (10)。 6. The method for detecting fake reviews based on BERT-CNN according to claim 1, characterized in that, In step S2, the addition and normalization layer includes an Add&Norm layer. Add adds the input and output of the self-attention layer (residual connection), and LayerNorm performs layer normalization, normalizing the hidden layers in the neural network to a standard normal distribution, thereby accelerating convergence.

7. The method for detecting fake reviews based on BERT-CNN according to claim 1, characterized in that, In step S2, the feedforward network layer consists of two fully connected layers plus a ReLU activation function, as shown in equation (11): FFN(x)=max(0,xW1+b1)W2+b2 (11) Where W1 and b1 represent the weights and biases of the first fully connected layer, respectively, and W2 and b2 represent the weights and biases of the second fully connected layer, respectively.

8. The method for detecting fake reviews based on BERT-CNN according to claim 1, characterized in that, In step S3, the convolutional neural network includes an embedding layer, a convolutional layer, a pooling layer, and a fully connected layer; In the embedding layer, the text word vectors trained by BERT are combined into an n×m matrix Z = [W1,…,W1]. n ], where W i The word vectors output by BERT; Convolutional layers extract convolutional features from the input text. Each convolutional kernel is like a window. By continuously moving the window, semantic learning is performed on the input text to extract different semantic features. The convolution kernel is convolved with the input matrix, and the formula for convolution is shown in equation (12): S=f(WZ+b)(12) Where S represents the feature matrix after the convolution operation, W represents the weight matrix, and b represents the offset vector; After convolution, feature mapping is required. Each convolutional layer contains multiple feature maps, and the mapping operation is performed by the ReLU function, as shown in equation (13): ReLU = max(0, x) (13) The pooling layer serves to select and filter features, simplifying computational complexity and extracting key features. The role of the fully connected layer is to combine the extracted features, use the learned features to achieve the learning objective, and send the output value to the classifier.

9. A method for detecting fake comments based on BERT-CNN according to any one of claims 1 to 8, characterized in that, In step S4, the softmax classifier uses the softmax function to calculate the probability. To prevent data overflow, the largest output value is subtracted from each output value, as shown in equations (14) and (15): D = max(Z) (14) Where Z = [X] [CLS] ,Z1,…,Z n ], X [CLS] The output after training with BERT is [Z1,…,Z]. n [Z1, ..., Zn] represents the feature matrix extracted by CNN after BERT training, where D is [Z1, ..., Zn]. n The maximum value in ]; Finally, the cross-entropy function is used to continuously update and optimize the parameters of the model, as shown in equation (16): Where C represents the size of the training set, y c p represents the tag value of the comment. c The model classification probability for the comments.

Images