Semantic analysis method based on natural language processing
By introducing a multi-level semantic analysis method that incorporates pre-trained word vectors, bidirectional LSTM, attention mechanisms, and Transformer self-attention mechanisms, this approach overcomes the limitations of traditional semantic analysis, achieving more efficient semantic understanding and context awareness, and is suitable for applications such as intelligent customer service and information retrieval.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- BEIJING UNION UNIVERSITY
- Filing Date
- 2024-11-04
- Publication Date
- 2026-05-07
AI Technical Summary
Traditional semantic analysis methods struggle to effectively handle semantic ambiguity, lack understanding of long-distance dependencies, rely on feature engineering and single-task learning, resulting in insufficient accuracy and generalization ability in semantic understanding.
We employ a pre-trained word vector model, a bidirectional LSTM network, an attention mechanism, and a self-attention mechanism based on the Transformer architecture, combined with classifiers and clustering algorithms, to perform multi-level semantic analysis and capture the deep features and long-distance dependencies of the text.
It improves the accuracy of semantic understanding and the generalization ability of the model, enabling it to capture semantic information in text more accurately, and is suitable for scenarios such as intelligent customer service and information retrieval.
Smart Images
Figure CN2024129699_07052026_PF_FP_ABST
Abstract
Description
A semantic analysis method based on natural language processing Technical Field
[0001] This invention relates to the field of semantic analysis technology, and in particular to a semantic analysis method based on natural language processing. Background Technology
[0002] Natural Language Processing (NLP) is an important research area in artificial intelligence, aiming to enable computers to understand, interpret, and generate natural language. With the rapid development of the internet and social media, the generation of massive amounts of text data has made effective semantic analysis of text increasingly important. Traditional semantic analysis methods mainly rely on rule-based techniques and shallow statistical models, which have the following limitations:
[0003] (1) Semantic ambiguity: Many words have different meanings in different contexts. Traditional methods are difficult to effectively deal with semantic ambiguity, which leads to a decrease in the accuracy of semantic understanding.
[0004] (2) Missing context: Traditional word vector models (such as Word2Vec and GloVe) usually only consider local context and cannot capture long-distance dependencies, which limits the model's ability to understand complex semantics.
[0005] (3) Feature engineering dependency: Many traditional methods rely on manually designed features, resulting in insufficient generalization ability of the model, and the features need to be redesigned when dealing with new domains or new tasks.
[0006] (4) Single-task learning: Most existing technologies focus on a single task (such as sentiment analysis or topic modeling), lack comprehensive consideration of multiple tasks, and cannot make full use of the correlation between different tasks.
[0007] In recent years, the rapid development of deep learning technology has brought new opportunities to natural language processing. In particular, deep learning-based models (such as LSTM, GRU, and Transformer) have performed exceptionally well in semantic analysis tasks, automatically learning deep features of text and significantly improving the accuracy of semantic understanding.
[0008] Despite the significant progress made by deep learning in semantic analysis, several challenges remain, such as model interpretability, dependence on limited labeled data, and high computational resource requirements. Therefore, developing a multi-layered semantic analysis method that combines the advantages of deep learning can better address the shortcomings of existing technologies and has significant research value and application prospects.
[0009] Summary of the Invention
[0010] In order to overcome the shortcomings of the prior art, the purpose of this invention is to provide a semantic analysis method based on natural language processing, which can more accurately capture semantic information in text.
[0011] To achieve the above objectives, the present invention provides the following solution:
[0012] A semantic analysis method based on natural language processing includes:
[0013] The text to be processed is cleaned and segmented to obtain word units;
[0014] Each word is converted into a vector representation using a pre-trained word vector model;
[0015] Word vectors are encoded using a bidirectional LSTM network to generate sentence-level semantic representations;
[0016] We use an attention mechanism to weightedly combine sentence-level representations to generate paragraph-level semantic representations;
[0017] The self-attention mechanism of the Transformer architecture is used to capture long-distance dependencies between the semantic representations at the sentence level and the semantic representations at the paragraph level, and the semantic similarity between the texts to be processed is determined based on the long-distance dependencies.
[0018] The text to be processed is analyzed for sentiment using a classifier, and the analysis results are obtained.
[0019] The topic structure in the text to be processed is identified using a clustering algorithm.
[0020] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:
[0021] This invention provides a semantic analysis method based on natural language processing. The method includes: cleaning and segmenting the text to be processed to obtain word units; converting each word into a vector representation using a pre-trained word vector model; encoding the word vectors through a bidirectional LSTM network to generate sentence-level semantic representations; weighting and combining the sentence-level representations using an attention mechanism to generate paragraph-level semantic representations; capturing long-distance dependencies between the sentence-level and paragraph-level semantic representations using a Transformer architecture's self-attention mechanism, and determining the semantic similarity between the texts to be processed based on the long-distance dependencies; and performing sentiment analysis on the texts to be processed using a classifier to obtain the analysis results. This invention, by introducing multi-layered semantic representations and context-aware mechanisms, can more accurately capture semantic information in text. Attached Figure Description
[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0023] Figure 1 is a flowchart of the method provided in an embodiment of the present invention. Detailed Implementation
[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0025] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0026] As shown in Figure 1, this invention provides a semantic analysis method based on natural language processing, including:
[0027] The text to be processed is cleaned and segmented to obtain word units;
[0028] Each word is converted into a vector representation using a pre-trained word vector model;
[0029] Word vectors are encoded using a bidirectional LSTM network to generate sentence-level semantic representations;
[0030] We use an attention mechanism to weightedly combine sentence-level representations to generate paragraph-level semantic representations;
[0031] The self-attention mechanism of the Transformer architecture is used to capture long-distance dependencies between the semantic representations at the sentence level and the semantic representations at the paragraph level, and the semantic similarity between the texts to be processed is determined based on the long-distance dependencies.
[0032] The text to be processed is analyzed for sentiment using a classifier, and the analysis results are obtained.
[0033] The topic structure in the text to be processed is identified using a clustering algorithm.
[0034] Preferably, the text to be processed is subjected to text cleaning and word segmentation to obtain word units, including:
[0035] Noise, punctuation marks, and stop words are removed from the text to be processed to obtain the cleaned text;
[0036] The cleaned text is segmented into words or sub-word units to obtain the word units.
[0037] Preferably, the formula for the sentence-level semantic representation is:
[0038] The parameters are explained below:
[0039] Parameter Description
[0040] X: The input sequence of word vectors, containing the vector representation of each word in the sentence.
[0041] x t The word vector of the t-th word is usually obtained through a pre-trained word vector model (such as Word2Vec or GloVe).
[0042] T: Sentence length, i.e., the number of words in the sentence.
[0043] h t The hidden state at time step t represents the sentence-level semantic representation at that time step.
[0044] LSTM forward : Forward LSTM network, which processes the input sequence from left to right.
[0045] The hidden state of the forward LSTM at time step t-1 represents the context information of the previous time step.
[0046] LSTM backward Backward LSTM network, which processes input sequences from right to left.
[0047] The hidden state of the backward LSTM at time step t+1 represents the context information of the next time step.
[0048] Preferably, the formula for the semantic representation at the paragraph level is:
[0049] Wherein, attention weight α i Calculated using the following formula:
[0050] And e i It is calculated using a scoring function, expressed as: e i =score(h i ,v)
[0051] The parameters are explained below:
[0052] H: A set of sentence-level representations, containing the semantic representation of each sentence in the paragraph.
[0053] h i The representation of the i-th sentence is usually the hidden state generated by a model such as LSTM or GRU.
[0054] n: The number of sentences in the paragraph.
[0055] P: Paragraph-level semantic representation is a weighted combination of all sentence representations.
[0056] α i The attention weight of the i-th sentence represents the importance of that sentence in generating the paragraph representation.
[0057] e i The score of the i-th sentence reflects its relevance to the context vector v.
[0058] v: Context vector, usually a fixed vector that represents the overall context information of the paragraph, or a specific vector (such as the average of sentence-level representations or the representation of a specific sentence).
[0059] Preferably, the formula for calculating the semantic similarity is:
[0060] The parameters are explained below:
[0061] S(P a ,P b ): The semantic similarity between text a and text b, with values ranging from -1 to 1, where 1 indicates complete similarity and -1 indicates complete dissimilarity.
[0062] P a The paragraph-level semantic representation of text a is usually a vector generated through a self-attention mechanism.
[0063] P b The paragraph-level semantic representation of text b is usually a vector generated through a self-attention mechanism.
[0064] P a ·P b : indicates P a and P b The dot product (inner product) is used to calculate the similarity between two vectors.
[0065] ||P a ||: The paragraph-level norm (L2 norm) of text 'a', calculated using the following formula: Where d is the dimension of the vector.
[0066] ||P b ||: The paragraph-level norm (L2 norm) of text b, calculated using the following formula:
[0067] Corresponding to the above method, this embodiment also provides a semantic analysis system based on natural language processing, including:
[0068] The word cleaning and segmentation unit is used to perform text cleaning and word segmentation on the text to be processed, and obtain word units.
[0069] The vector transformation unit is used to convert each word into a vector representation using a pre-trained word vector model;
[0070] The first semantic representation unit is used to encode word vectors through a bidirectional LSTM network to generate sentence-level semantic representations;
[0071] The second semantic representation unit is used to weight and combine sentence-level representations using an attention mechanism to generate paragraph-level semantic representations;
[0072] The similarity calculation unit is used to capture the long-distance dependencies between the semantic representations at the sentence level and the semantic representations at the paragraph level using the self-attention mechanism of the Transformer architecture, and to determine the semantic similarity between the texts to be processed based on the long-distance dependencies.
[0073] The sentiment analysis unit is used to perform sentiment analysis on the text to be processed using a classifier to obtain the analysis results.
[0074] The structure recognition unit is used to identify the topic structure in the text to be processed through a clustering algorithm.
[0075] The beneficial effects of this invention are as follows:
[0076] This invention, by introducing multi-layered semantic representation and context-aware mechanisms, can more accurately capture semantic information in text. Compared to existing technologies, this invention represents a significant innovation in the refinement of semantic representation and the depth of context understanding. This invention has broad application prospects in the field of natural language processing and can be used in various scenarios such as intelligent customer service, information retrieval, and sentiment analysis.
[0077] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0078] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A semantic analysis method based on natural language processing, characterized in that, include: The text to be processed is cleaned and segmented to obtain word units; Each word is converted into a vector representation using a pre-trained word vector model; Word vectors are encoded using a bidirectional LSTM network to generate sentence-level semantic representations; We use an attention mechanism to weightedly combine sentence-level representations to generate paragraph-level semantic representations; The self-attention mechanism of the Transformer architecture is used to capture long-distance dependencies between the semantic representations at the sentence level and the semantic representations at the paragraph level, and the semantic similarity between the texts to be processed is determined based on the long-distance dependencies. The text to be processed is analyzed for sentiment using a classifier to obtain the analysis results. The topic structure in the text to be processed is identified using a clustering algorithm.
2. The semantic analysis method based on natural language processing according to claim 1, characterized in that, The text to be processed is cleaned and segmented to obtain word units, including: Noise, punctuation marks, and stop words are removed from the text to be processed to obtain the cleaned text; The cleaned text is segmented into words or sub-word units to obtain the word units.
3. The semantic analysis method based on natural language processing according to claim 1, characterized in that, The formula for the sentence-level semantic representation is: Where X is the input word vector sequence, containing the vector representation of each word in the sentence, x t Let h be the word vector of the t-th word, T be the length of the sentence (i.e., the number of words in the sentence), and h be the word vector of the t-th word. t For the hidden state at time step t, representing the sentence-level semantic representation at that time step, LSTM... forward This is a feedforward LSTM network that processes the input sequence from left to right. The hidden state of the forward LSTM at time step t-1 represents the context information of the previous time step. backward For a backward LSTM network, it processes the input sequence from right to left. This represents the hidden state of the backward LSTM at time step t+1, indicating the context information of the next time step.
4. The semantic analysis method based on natural language processing according to claim 1, characterized in that, The formula for the semantic representation at the paragraph level is: Where, α i Let i be the attention weight for the i-th sentence. e i Let e be the score for the i-th sentence. i =score(h i H is a set of sentence-level representations, containing the semantic representation of each sentence in the paragraph. i Let represent the i-th sentence, n be the number of sentences in the paragraph, P be the paragraph-level semantic representation, and v be the context vector.
5. The semantic analysis method based on natural language processing according to claim 1, characterized in that, The formula for calculating semantic similarity is: Among them, S(P a ,P b P represents the semantic similarity between text a and text b, with values ranging from -1 to 1, where 1 indicates complete similarity and -1 indicates complete dissimilarity. a P is a paragraph-level semantic representation of the text to be processed, typically a vector generated through a self-attention mechanism. b For the paragraph-level semantic representation of the text b to be processed, ||P a || is the norm of the paragraph-level representation of the text 'a' to be processed, calculated using the following formula: Where d is the dimension of the vector, ||P b || is the norm of the paragraph-level representation of the text b to be processed, calculated using the following formula: