Determination method and apparatus for feature representation, device, and storage medium

By converting text units into weight vectors and using these weight vectors to represent feature importance, and combining key vectors and value vectors to calculate attention and feature representations, the high computational complexity of self-attention mechanisms is solved, achieving more efficient feature representation and semantic capture.

WO2026233355A1PCT designated stage Publication Date: 2026-11-12CHINA UNITED NETWORK COMM GRP CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2026/095162
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-07
Filing Date
2026-05-07
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

Existing self-attention mechanisms perform global uniform computation on all text units in natural language processing classification scenarios, resulting in high computational complexity and reduced model computational efficiency, especially the significant impact of redundant computation on key feature text units.

Method used

By converting text units into weight vectors and using these weight vectors to represent feature importance, attention representations and feature representations are calculated by combining key vectors and value vectors, reducing dependence on query vectors and lowering computational complexity.

Benefits of technology

The attention weight calculation process has been simplified, improving the computational efficiency of the model in classification scenarios. It focuses on key features, reduces redundant calculations, and enhances the model's computational efficiency and semantic capture capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2026095162_12112026_PF_FP_ABST
    Figure CN2026095162_12112026_PF_FP_ABST
Patent Text Reader

Abstract

The present application provides a determination method for a feature representation. The method comprises: acquiring a first text unit, and performing vectorization processing on the first text unit to acquire a vector corresponding to the first text unit, the first text unit being any text unit obtained by segmenting a target text; on the basis of the vector, determining a key vector, a value vector, and a weight vector corresponding to the first text unit; on the basis of the key vector and the value vector corresponding to the first text unit, determining an attention representation corresponding to the first text unit; on the basis of the vector, the key vector, and the value vector corresponding to the first text unit, determining a semantic state value corresponding to the first text unit, the semantic state value being used for representing the degree of semantic association between the first text unit and text units preceding the first text unit; and on the basis of the attention representation and the semantic state value corresponding to the first text unit, determining a feature representation of the first text unit.
Need to check novelty before this filing date? Find Prior Art

Description

A method, apparatus, device, and storage medium for determining feature representation

[0001] This application claims priority to Chinese Patent Application No. 202510585160.5, filed on May 7, 2025, entitled “A method, apparatus, device and storage medium for determining feature representation”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of natural language processing technology, and in particular to a method, apparatus, device and storage medium for determining feature representation. Background Technology

[0003] In the field of natural language processing, the Transformer architecture, with its parallel computing power and long-distance dependency modeling capabilities, has become the core architecture for many large language models. Its self-attention mechanism generates context-dependent feature representations by calculating the association weights between text units in the input text, thereby encoding the semantics of the text. Summary of the Invention

[0004] This application provides a method, apparatus, device, and storage medium for determining feature representation.

[0005] Firstly, this application provides a method for determining feature representations. The method includes: acquiring a first text unit and vectorizing the first text unit to obtain a vector corresponding to the first text unit, wherein the first text unit is any text unit obtained by segmenting a target text. Based on the vector, determining the key vector, value vector, and weight vector corresponding to the first text unit, where the weight vector represents the feature importance of the first text unit. Based on the key vector and value vector corresponding to the first text unit, determining the attention representation corresponding to the first text unit. Based on the vector, key vector, and value vector corresponding to the first text unit, determining the semantic state value corresponding to the first text unit. The semantic state value represents the degree of semantic association between the first text unit and previous text units. Based on the attention representation and semantic state value corresponding to the first text unit, determining the feature representation of the first text unit.

[0006] One possible implementation involves determining the attention representation of a first text unit based on its key vector and value vector. This includes: obtaining at least one second text unit based on its index position in the target text, where the index of each second text unit in the target text is less than or equal to the index of the first text unit in the target text; obtaining the weight vector and value vector corresponding to each second text unit; normalizing the weight vectors of each second text unit to determine its attention weight; and determining the attention representation of the first text unit based on its attention weight and value vector.

[0007] One possible implementation involves normalizing the weight vectors corresponding to each second text unit to determine the attention weights for each second text unit, including:

[0008] The attention weights for each second text unit are determined using the following formula:

[0009] Where, α i w represents the attention weight corresponding to the i-th second text unit. i Let represent the weight vector corresponding to the i-th second text unit, t represent the total number of second text units, and exp represent the exponential operation on the weight vector.

[0010] One possible implementation involves determining the attention representation of the first text unit based on the attention weights and value vectors of each second text unit, including: weighted summation of the value vectors of each second text unit based on their attention weights to obtain the attention representation of the first text unit.

[0011] One possible implementation involves weighted summation of the value vectors of each second text unit based on their attention weights to obtain the attention representation of the first text unit, including:

[0012] The attention representation of the first text unit is obtained using the following formula:

[0013] Among them, o t The attention representation of the first text unit, α i v represents the attention weight corresponding to the i-th second text unit. i Let represent the value vector corresponding to the i-th second text unit, and t represent the total number of second text units.

[0014] One possible implementation involves determining the semantic state value of a first text unit based on its corresponding vector, key vector, and value vector. This includes: obtaining a third text unit based on its index position in the target text, where the third text unit is the preceding text unit of the first text unit in the target text; obtaining the semantic state value of the third text unit; multiplying the key vector and value vector element-wise to obtain the key-value interaction vector of the first text unit; and determining the semantic state value of the first text unit based on the semantic state value of the third text unit, the key-value interaction vector of the first text unit, and the vector.

[0015] One possible implementation involves determining the semantic state value of the first text unit based on the semantic state value of the third text unit, the key-value interaction vector of the first text unit, and a vector, including: determining the semantic state value of the third text unit according to the following formula: s t =βs t-1 +(1-β)(k t v t )+h t

[0016] Among them, s t s represents the semantic state value corresponding to the first text unit. t-1 β represents the semantic state value corresponding to the third text unit, and k represents the weight of the semantic state value corresponding to the third text unit. t v t Let h represent the key-value interaction vector of the first text unit, 1-β represent the weight of the key-value interaction vector of the first text unit, and h represent the value of the key-value interaction vector of the first text unit. t This represents the text unit vector corresponding to the first text unit.

[0017] One possible implementation involves determining the feature representation of a first text unit based on its attention representation and semantic state value. This includes: performing a linear transformation on the attention representation of the first text unit; and adding the result of the linear transformation to the semantic state value to determine the feature representation of the first text unit.

[0018] Secondly, this application provides a feature representation determination apparatus, which includes various functional modules for the method described in the first aspect above.

[0019] Thirdly, this application provides an electronic device comprising a processor and a memory. The memory stores processor-executable instructions, which, when configured to execute the instructions, cause the electronic device to perform the method described in the first aspect above.

[0020] Fourthly, this application provides a readable storage medium comprising software instructions. When the software instructions are executed in an electronic device, they cause the electronic device to perform the method described in the first aspect above. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 is a flowchart illustrating a method for determining a feature representation according to an embodiment of this application;

[0023] Figure 2 is a flowchart illustrating a method for determining a weight vector according to an embodiment of this application;

[0024] Figure 3 is a flowchart illustrating a method for determining attention representation provided in an embodiment of this application;

[0025] Figure 4 is a flowchart illustrating a method for determining semantic state values ​​provided in an embodiment of this application;

[0026] Figure 5 is a flowchart illustrating a method for determining feature representation based on linear transformation provided in an embodiment of this application;

[0027] Figure 6 is a schematic diagram of the composition of a feature representation determination device provided in an embodiment of this application;

[0028] Figure 7 is a schematic diagram of the composition of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] Hereinafter, the terms "first," "second," and "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first," "second," or "third," etc., may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0030] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0031] To facilitate understanding, the technical terms involved in the embodiments of this application will be introduced first.

[0032] A text unit, also known as a token, is the basic unit in natural language processing. It can be a phrase, a word, or even a character. When large language models process text, they typically need to segment the text into individual text units so that the computer can understand and process it. For example, the sentence "I like natural language processing" might be segmented into text units such as "I," "like," "natural," "language," and "processing." These text units are encoded and transformed, and then embedded to obtain corresponding vector representations. These vector representations serve as the foundational data units for subsequent model processing of text semantics and syntax, as well as for performing various tasks, helping the model to uncover deeper information from the text.

[0033] Self-attention is the core computational unit of the Transformer, used to measure the degree of correlation between various text units in the input sequence and generate context-sensitive feature representations. Its computation process is as follows:

[0034] 1. Divide the input text into multiple text units and convert each text unit into a vector.

[0035] 2. Perform three different linear transformations on each vector to obtain the query vector (Q), key vector (K), and value vector (V).

[0036] 3. For each text unit's query vector, calculate its dot product with the key vectors of all other text units to obtain the attention score of that text unit relative to all other text units. This process can be represented as:

[0037] Where, d k It is the dimension of the key vector K, divided by This is to prevent the dot product result from becoming too large, which would cause the gradient of the softmax function to vanish.

[0038] 4. Normalize the attention score of each text unit using the softmax function to obtain the attention weight of each text unit. Then, sum the value vector of each text unit using its attention weight to obtain the feature representation of each text unit. This process can be represented as:

[0039] Where n is the total number of text units in the input text, and score i V represents the attention score of the current text unit relative to the i-th text unit. j This represents the value vector of the j-th text unit.

[0040] It should be understood that, as shown in formula (1), for n text units, the query vector corresponding to each text unit needs to be multiplied by the key vectors of all the other n text units to calculate the attention score. This process needs to be performed n×n times in total. Therefore, the computational complexity of this attention score is O(n^2). 2 ).

[0041] In text classification scenarios using large language models, the core semantics of the input text are typically determined by a subset of text units with key features, while the remaining text units, although containing semantic information, have a smaller impact on the classification result. However, existing self-attention mechanisms in classification tasks employ a global computation strategy to perform correlation calculations on all text units. This leads to redundant computations on non-critical text units, increasing the model's computational complexity (O(n)). 2 This reduces the computational efficiency of the model.

[0042] As mentioned above, in classification scenarios, some text units in the input text possess key features that distinguish text categories, and these key features often determine the text's classification. For example, in sentiment analysis tasks, emotional polarity words such as "like," "dislike," "praise," and "criticism" serve as key text units, significantly influencing whether the text is classified as positive, negative, or neutral. Existing self-attention mechanisms typically perform global uniform computation on all text units, modeling the interaction of all text unit pairs through fully connected attention weight allocation. However, in classification scenarios, this approach ignores the decisive role of key feature text units, resulting in redundant computation of less critical text units and ultimately impacting the model's computational efficiency.

[0043] Therefore, how to calculate the feature representation of text units by considering the key features of each text unit in the input text in a classification business scenario has become an urgent problem for us to solve.

[0044] This application provides a method for determining feature representations. By converting each text unit into a weight vector, the key features of each text unit are visualized. Furthermore, the attention weight of each text unit is calculated using the weight vector, and the attention representation and feature representation of each text unit are calculated by combining the value vector and the key vector. This reduces the complexity of the model in feature representation calculation and improves the computational efficiency of the model in classification scenarios.

[0045] First, a brief introduction to the application scenarios of the embodiments of this application will be given.

[0046] The feature representation determination method provided in this application can be applied to computing devices that are used for model training tasks. The computing device can be a server cluster consisting of multiple servers, a single server, a computer, or a processor or processing chip within a server or computer. This application does not limit the specific form of the computing device.

[0047] The method for determining feature representations provided in the embodiments of this application will now be described in conjunction with the accompanying drawings.

[0048] As shown in Figure 1, when this method is applied to a computing device, it includes the following:

[0049] S101. Obtain the first text unit and perform vectorization processing on the first text unit to obtain the vector corresponding to the first text unit.

[0050] The first text unit is any text unit obtained by segmenting the target text.

[0051] In some embodiments, the method for segmenting target text into multiple text units may include: simple word segmentation based on spaces and punctuation marks, maximum forward matching or reverse matching based on dictionary matching, word segmentation method based on statistical models or deep learning, etc. The specific segmentation method is not limited in the embodiments of this application.

[0052] Specifically, simple word segmentation methods based on spaces and punctuation are suitable for texts like English where words are separated by spaces. Maximum forward matching or backward matching methods based on dictionary matching are suitable for languages ​​like Chinese without explicit delimiters, segmenting words using a predefined dictionary. Word segmentation methods based on statistical models or deep learning, such as those using Hidden Markov Models, Conditional Random Fields, or Bidirectional Long Short-Term Memory Networks, learn word boundary patterns through training data, achieving more intelligent text segmentation.

[0053] In some embodiments, the method for vectorizing the first text unit includes word embedding encoding, dictionary encoding, and word encoding.

[0054] For example, the specific process of vectorizing the first text unit using dictionary encoding is as follows:

[0055] First, the computing device needs to pre-build a dictionary covering all unique words in the target text corpus and assign a unique index to each word. Next, after preprocessing the first text unit (removing punctuation, standardizing size, etc.), it is split into multiple words according to the dictionary, and these words are mapped to their corresponding indices in the dictionary. If a word in the first text unit does not have a corresponding index in the dictionary, it is mapped to a specific index (which can be understood as an identifier for the unknown word, its value being the maximum index value in the dictionary plus 1). Specifically, the specific index allows the model to treat the unknown word as an element with some fuzzy semantics, referring to its information in a relatively conservative way during the classification decision process. Then, based on the index of each word, they are assembled into an index sequence, and the index sequence is adjusted to a fixed length using padding or truncation operations. Finally, the processed index sequence is converted into a vector using one-hot encoding or other methods.

[0056] S102. Based on vectors, determine the key vector, value vector, and weight vector corresponding to the first text unit.

[0057] The weight vector is used to represent the feature importance of the first text unit.

[0058] Specifically, key vectors are used to measure the relevance between text units. Essentially, key vectors construct a "searchable" feature space, helping the model identify which contextual information requires priority. In existing attention mechanisms, the dot product of the query vector and the key vector is often used to determine the correlation strength between the current text unit and other text units. However, in this embodiment, the key vector is primarily used in the calculation of the first text semantic state value, while weight vectors are used to measure the feature importance of the current text.

[0059] Value vectors are used to carry the actual semantic content of text units. After the attention weights are calculated, the value vectors are summed according to their weights to generate the attention representation. Unlike key vectors, value vectors do not participate in relevance calculations; instead, they provide the raw information that needs to be aggregated. This design ensures that after attention assignment, the model can fuse the most relevant semantic features, rather than relying solely on relevance scores.

[0060] Weight vectors are used to represent the feature importance of a text unit. As is known from the background technology, there are some text units in the target text that play a key role in text classification. Through these text units, the classification category of the target text can be basically determined. This application, by converting text units into weight vectors, can visualize the feature importance of text units, allowing computing devices to directly calculate the relevance (attention weight) of text units in the target text based on the weight vectors of the text units. This reduces the dependence on query vectors in existing self-attention mechanisms and lowers the computational complexity of the interaction between query vectors and value vectors.

[0061] One possible implementation is that the vector can be obtained by performing a linear transformation on the key vector matrix to obtain the corresponding key vector, and by performing a linear transformation on the value vector matrix to obtain the corresponding value vector. Specifically, this process can be referred to existing self-attention mechanisms, which will not be elaborated here.

[0062] It should be noted that the key vector transformation matrix extracts the location index features (such as syntactic structure and topic category) of the data through dimensionality reduction for fast retrieval; the value vector transformation matrix maps the data to a high dimension, fully preserving the original semantic details and entity attributes, providing information for deep semantic analysis; the weight vector transformation matrix identifies discriminative features (such as classification and sensitive information) during model training, maps them to an independent dimensional space, and improves the model's ability and performance to process specific features.

[0063] It should also be noted that while the parameters of these three vector matrices are independent, their matrix structures (e.g., dimensions, parameter magnitudes) are identical, and all are linear transformations achieved through matrix multiplication with vectors. Therefore, the final output sequences of key vectors, value vectors, and weight vectors from these matrices all share the same dimensions and parameter magnitudes, providing a unified data format and computational foundation for subsequent attention representation and semantic state value determination.

[0064] The weight vector directly represents the feature importance of a text unit. The attention weight distribution can be obtained by normalizing its values. This design allows computing devices to directly calculate the relevance of text units in the target text based on the weight vector, reducing the dependence of traditional self-attention mechanisms on the query vector. Specifically, in text classification tasks, the key text units contained in the target text explicitly quantify their feature importance through the weight vector, enabling the model to effectively focus on these decisive features. This method simplifies the calculation process of attention weights, reduces computational complexity, and makes classification decisions more direct and efficient. Maintaining the same dimension as the value vector ensures that the normalized weights can be directly used for weighted aggregation of value vectors, ultimately generating a semantic representation with prominent key points, improving computational efficiency while preserving semantic capture capabilities.

[0065] S103. Based on the key vector and value vector corresponding to the first text unit, determine the attention representation corresponding to the first text unit.

[0066] The attention representation can also be called the attention output vector.

[0067] Attention representation refers to the resulting vector obtained after attention calculation. In this embodiment, the attention representation of the first text unit can reflect the degree of contribution of its (classification) features to the semantics of the text. During the calculation of attention representations of multiple text units in the target text, the first text unit with a higher attention representation has a greater determining effect on the semantics of the text, enabling the model to automatically focus on key semantic information and capture the core content and key features of the text more quickly.

[0068] Specifically, the process first requires calculating the attention weights corresponding to the first text unit based on its weight vector. Then, based on these attention weights, the attention representation of the first text unit is calculated.

[0069] In some embodiments, the computing device can jointly calculate the attention weights and attention representations of the first text unit by combining text units in the target text whose positions are less than or equal to the first text unit. The core of this approach is that the weight vector itself encodes the feature importance of each text unit. By applying it to context units within a defined range (i.e., text units whose index is less than or equal to the current unit), two key effects can be achieved: First, it maintains the causality of sequence processing, ensuring that the representation of each text unit depends only on current and historical information. Second, through the dynamic filtering of historical information by the weight vector, it highlights key contexts that actually contribute to the current unit. This design retains the advantage of traditional self-attention in capturing long-distance dependencies while improving the targeting and efficiency of attention calculation through prior importance filtering by the weight vector.

[0070] Specifically, the specific process of this embodiment can be described in the following figure 3 corresponding to the embodiment, which will not be elaborated here.

[0071] S104. Based on the vector, key vector and value vector corresponding to the first text unit, determine the semantic state value corresponding to the first text unit.

[0072] The semantic state value is used to represent the degree of semantic association between the corresponding text unit and the previous text unit.

[0073] It should be noted that the semantic state value is obtained by calculating the semantic information of the current text unit and the semantic information of the previous text unit. This semantic state value can represent the degree of semantic integration of the current text unit in the context formed by the previous text units.

[0074] For example, taking machine translation as an example, suppose we need to translate the sentence "I like to eat apples, apples are my favorite fruit." When processing the second text unit "apple," its semantic state value reflects not only the semantic information of "apple" itself, but also the semantic dependencies accumulated from the preceding "I like to eat apples," demonstrating the coherence and progression of the text's semantics. Its role is to provide the model with a dynamically updated semantic representation, enabling the model to perform more accurate semantic understanding and expression based on the accumulated semantic information when processing each text unit.

[0075] Specifically, for the specific implementation of step S104, please refer to the description of the embodiment corresponding to Figure 4 below, which will not be elaborated here.

[0076] S105. Based on the attention representation and semantic state value corresponding to the first text unit, determine the feature representation of the first text unit.

[0077] The attention representation is obtained based on the weight vector and value vector of the text unit, and the value is in vector form. The semantic state value is obtained based on the key vector, value vector and text unit vector of the text unit, and the value is also in vector form.

[0078] It should be noted that the semantic state value can supplement and enhance the attention representation obtained by calculating the weight vector, so that the feature representation not only focuses on the local features of the text unit, but also takes into account the dynamic changes of the entire text context.

[0079] For example, in sentiment analysis tasks, for the sentence "The movie's pacing was slow and disappointing in the beginning, but the ending was wonderful and surprising," ordinary attention mechanisms might only assign weights based on simple associations between words. However, by combining semantic state values, when processing "but the ending was wonderful and surprising," it can more accurately adjust the attention weights for words such as "wonderful" and "surprising" based on the semantic state constructed by the preceding "disappointment," reducing the risk of semantic information loss or misjudgment caused by relying solely on attention weights.

[0080] It should also be noted that although the attention representation and semantic state value have the same vector dimension (i.e., vector length), they are generated through different computational paths (attention representation is based on weight vectors and value vectors, while semantic state values ​​are based on key vectors, value vectors, and text unit vectors), meaning that the feature space distributions they represent differ. Therefore, when determining the feature representation of the first text unit based on the attention representation and semantic state value, a linear transformation or feature projection can be used to map them to a unified feature space before performing a fusion operation to ensure the effectiveness of the feature representation.

[0081] In some embodiments, the computing device may use methods such as linear projection fusion, gated fusion, feature splicing + fully connected layer fusion to map the attention representation and semantic state value of the first text unit to a unified feature space, and then determine the feature representation of each text unit based on the two.

[0082] The feature representation determination method provided in this application transforms the vector corresponding to a text unit into a weight vector. The weight vector represents the importance of the (classification) features of the text unit, and text units with key features are transformed into high-value weight vectors. Furthermore, attention representation is calculated based on the weight vector of the first text unit. Compared to existing self-attention mechanisms that rely on the interaction between query vectors and key vectors between text units, this application directly focuses on key features, reducing the number of indiscriminate calculations on all text units in the target text. This effectively reduces the computational complexity of the self-attention mechanism and improves the computational efficiency of large language models for input text feature representation. Simultaneously, by combining attention representation with semantic state values ​​to determine feature representation, it utilizes weight vectors to highlight key features while preserving the contextual semantic information of text units, achieving an effective balance between computational efficiency and semantic expressive power.

[0083] The following section, in conjunction with the accompanying diagram, explains how to obtain the weight vector corresponding to the first text unit.

[0084] In some embodiments, the computing device can multiply the learnable weight vector matrix with the vector corresponding to the first text unit to obtain the weight vector corresponding to the first text unit. In this case, the process is shown in Figure 2. Step S102 may further include the following steps:

[0085] S1021. Obtain the weight vector matrix.

[0086] The weight vector matrix is ​​used to convert a vector into a weight vector.

[0087] It should be noted that the weight vector matrix is ​​essentially a trainable parameter matrix. During model training, this parameter matrix can be dynamically adjusted using optimization algorithms (such as gradient descent) to automatically learn the optimal parameter configuration. After sufficient training, this vector matrix can be used to linearly transform the vector of the first text unit to obtain the weight vector, thus representing the importance of the most discriminative features in the first text unit for the current classification task.

[0088] In some embodiments, the weight vector matrix can be obtained in the following ways: random initialization training, pre-training transfer learning, and rule-guided learning.

[0089] For example, the specific process of obtaining the weight vector matrix through random initialization training is as follows:

[0090] First, a randomly initialized weight vector matrix is ​​generated according to a certain distribution (such as a Gaussian distribution). Then, training data with classification labels is input into the large language model. The model automatically identifies key features in the text unit vectors and then performs a linear mapping between these text unit vectors and the randomly initialized weight vector matrix to obtain the weight vector. The loss function value is calculated based on the difference between the model's prediction and the true labels to measure the prediction error. Then, an optimization algorithm (such as stochastic gradient descent) is used to update the elements of the weight vector matrix according to the gradient of the loss function, continuously adjusting the matrix parameters to reduce the loss function value. This training process is repeated iteratively, updating the weight vector matrix, until the model performance reaches a satisfactory level or a preset training stopping condition is met.

[0091] For example, the specific process of obtaining the weight vector matrix through rule-guided training is as follows:

[0092] First, an initial weight vector matrix is ​​constructed based on domain knowledge, experience, or specific rules. For example, in some natural language processing tasks, the initial values ​​of the matrix elements are set according to rules such as word part-of-speech and semantic relevance. Then, training data is input into the model. The model identifies key features of text unit vectors and uses these text unit vectors and the initial weight vector matrix to calculate the weight vector and the model's prediction results. During training, not only should the model's prediction error be considered, but additional rule constraints should be introduced into the loss function to ensure that the weight vector matrix elements conform to the pre-defined rules. By optimizing the algorithm to minimize the loss function while ensuring that the rule constraints are met, the weight vector matrix parameters are gradually adjusted, allowing the model to continuously improve its prediction performance while adhering to the rules, ultimately resulting in a weight vector matrix that conforms to the rule guidance.

[0093] It should be understood that the weight vector matrix can be continuously adjusted along with the training process of a large language model. This characteristic allows the weight vector matrix to dynamically adapt to diverse training data and complex task requirements. As training progresses, it can flexibly optimize parameters, more accurately identify key features in text unit vectors, and rationally allocate weights based on these features. When processing text with different semantic structures, the weight vector matrix adaptively adjusts the mapping rules, assigning higher weights to text unit vectors containing key classification features and weakening the weights of redundant or secondary information, helping the model to more efficiently focus on key classification features in the text.

[0094] S1022. Based on the vector and the weight vector matrix, determine the weight vector corresponding to the first text unit.

[0095] In some embodiments, the first text unit can be multiplied with the weight vector matrix, and the result can be used as the weight vector corresponding to the first text unit.

[0096] Specifically, in this calculation process, the model amplifies feature components strongly correlated with the classification objective (such as sentiment polarity features in sentiment analysis or medical terminology features in medical texts), while suppressing irrelevant or interfering feature components (such as the feature representation of grammatical function words or common stop words). This feature selection capability is achieved through backpropagation optimization on a large amount of training data. The model continuously adjusts the parameters of the vector matrix so that key text units can obtain more significant feature representations after transformation, thereby having a greater impact in subsequent attention calculations.

[0097] One possible implementation involves first treating the text unit vector as a row vector, multiplying it by each column element of the weight vector matrix, and then summing the results to obtain a numerical value. This numerical value is the element value at the corresponding position in the weight vector. Performing this operation on each column of the weight vector matrix yields a new vector, which is the weight vector obtained by linearly mapping the text unit vector through the weight vector matrix. By sequentially performing the above operation on each text unit vector in the text unit vector sequence, the weight vector corresponding to each text unit vector can be obtained.

[0098] As can be seen from steps S1021-S1022, the key features of the text unit vector are visualized as weight vectors through the dynamically learned weight vector matrix. This method can more directly quantify the feature importance of each text unit, provide accurate weight allocation for subsequent semantic state calculation, and effectively improve the model's efficiency in understanding text semantics.

[0099] The following section describes the acquisition of the attention representation for the first text unit, with reference to the accompanying diagram.

[0100] In some embodiments, the specific process of determining the attention representation of the first text unit is shown in Figure 3. Step S103 may specifically include the following steps:

[0101] S1031. Based on the index position of the first text unit in the target text, obtain at least one second text unit.

[0102] In this context, the index of each second text unit in the target text is less than or equal to the index of the first text unit in the target text.

[0103] In some embodiments, when segmenting the target text into multiple text units, the text units in the target text can be sorted based on their index positions within the target text. Then, during step S1031, the second text unit is determined based on the index position of the first text unit.

[0104] It should be noted that if the first text unit is the first text unit in the target text, then its attention is represented as its own value vector, and there is no need to perform steps S1032-S1033.

[0105] S1032. Obtain the weight vector and value vector corresponding to each second text unit.

[0106] It should be noted that the second text unit is the text unit in the target text segmentation text unit whose index position is less than or equal to that of the first text unit. The method for obtaining its weight vector and value vector is the same as that for obtaining the first text unit.

[0107] Specifically, to obtain the weight vector and value vector corresponding to each second text unit, each second unit can first be converted into a vector. Then, these vectors are linearly transformed with the weight vector matrix and the value vector matrix to obtain the weight vector and value vector corresponding to each second unit. This process can be referred to the embodiment of step S102 in Figure 1 above and the embodiment in Figure 2 above, and will not be described in detail here.

[0108] S1033. Normalize the weight vector corresponding to each second text unit to determine the attention weight of each second text unit.

[0109] It should be noted that in existing self-attention mechanisms, the query vector and key vector are used to calculate the attention score of a text unit through operations such as dot product. Then, the attention scores of each text unit are normalized to obtain the attention weight of each text unit, thereby measuring the degree of correlation between different text units. However, the weight vector in this application is essentially a quantified expression of the importance of the semantic features of a text unit, and its function can be seen as equivalent to the attention score in existing self-attention mechanisms. Therefore, the technical effect of calculating the attention weight of the first text unit using the weight vector is similar to the technical effect of calculating the attention weight based on the attention score.

[0110] It should be understood that using weight vectors to determine the attention weights of text units can reduce the redundant computation required by existing self-attention mechanisms, which involve performing dot product operations on other text units one by one. By directly weighting the text unit vectors with pre-calculated weight vectors, key information can be quickly focused. This approach transforms complex pairwise interactive computations into a more efficient linear mapping and weight allocation process, effectively reducing computational complexity and time overhead.

[0111] In some embodiments, the weight vectors corresponding to each second text unit can be normalized using the softmax function. Specifically, by processing the weight vectors with softmax, they can be converted into attention weights corresponding to each text unit (values ​​between 0 and 1, with a sum of 1), thereby determining the relative importance of different text units when calculating attention.

[0112] For example, the process of normalizing the weight vector corresponding to each second text unit to determine the attention weight of each second text unit among all second text units can be expressed as follows:

[0113] Where, α i w represents the attention weight corresponding to the i-th second text unit. i Let represent the weight vector corresponding to the i-th second text unit, t represent the total number of second text units, and exp represent the exponential function to perform an exponential operation on the weight vector. This operation can ensure that the final weight obtained is positive, and can also strengthen vectors with higher weight values ​​and weaken vectors with lower weight values.

[0114] For example, suppose the target text is divided into 4 text units, namely token1, token2, token3, and the first text unit is token3. Then the second text units are token1, token2, and token3, and the weight vectors of these second text units have a dimension of 2, namely w1 = [1, 0.5], w2 = [2, 1], and w3 = [3, 1.5].

[0115] Taking token1 as an example, the calculation process of its attention weight is expressed as follows: Then, calculate the values ​​for each element in the vector separately. Similarly, we can obtain the attention weights for token2 and token3: α2 = [0.24, 0.31] and α3 = [0.67, 0.051].

[0116] S1034. Based on the attention weights and value vectors of each second text unit, determine the attention representation of the first text unit.

[0117] In some embodiments, the computing device may perform a weighted summation of the value vectors of each second text unit based on the attention weights of each second text unit to obtain the attention representation of the first text unit.

[0118] For example, the weighted summation process can be represented as:

[0119] Among them, ot The attention representation of the first text unit, α i v represents the attention weight corresponding to the i-th second text unit. i Let represent the value vector corresponding to the i-th second text unit, and t represent the total number of second text units.

[0120] For example, suppose there are second text units named token1, token2, and token3, with a weight vector and value vector of dimension 2. Their weight vectors are α1 = [0.09, 019], α2 = [0.24, 0.31], and α3 = [0.67, 051], respectively, and their value vectors are v1 = [4, 2], v2 = [5, 3], and v3 = [6, 4]. Substituting the weight vector and value vector of each second text unit into formula (4) yields: o t = α1v1 + α2v2 + α3v3, and then during the multiplication of each weight vector and value vector, the elements at the same position in the vectors are multiplied, i.e., o t =[o t1 ,o t2 ]=[α 11 v 11 +α 12 v 12 +α 13 v 13 α 21 v 21 +α 22 v 12 +α 23 v 23 Substituting each element from the weight vector and value vector above, we get: o t =[0.09×4+0.24×5+0.67×6,0.19×2+0.31×3+0.51×4]=[5.58,3.35。

[0121] As can be seen from the above steps S1031-S1034, calculating the attention representation of the first text unit in the preceding text based on the weight vector can reduce the redundant calculation of each text unit performing dot product operations on other text units one by one in the existing self-attention mechanism. It transforms the complex pairwise interaction calculation into a linear superposition operation, which can effectively reduce the computational complexity and time overhead, and improve the efficiency and accuracy of the model in understanding the semantics of the text.

[0122] The following section describes how to obtain the semantic state value of the first text unit, with reference to the accompanying diagram.

[0123] In some embodiments, the specific process of determining the semantic state value of the first text unit is shown in Figure 4. Step S104 may specifically include the following steps:

[0124] S1041. Based on the index position of the first text unit in the target text, obtain the third text unit.

[0125] The third text unit is the text unit preceding the first text unit in the target text.

[0126] Specifically, this step can be referred to in step S1031, and will not be repeated here.

[0127] It should be noted that if the first text unit is the first text unit in the target text, since it does not have a corresponding third text unit, the semantic state value can be determined by calculating its own key-value interaction vector and the sum of the vectors (that is, when the first text unit is the first unit in the target text, there is no need to execute step S1042).

[0128] S1042. Obtain the semantic state value of the third text unit.

[0129] It should be noted that the semantic state value of the third text unit is also obtained by executing steps S1041-S1044 of this application. In the target text processing flow, the same logic is followed to determine the semantic state value of each text unit. When obtaining the semantic state value of the third text unit, the semantic state value of its predecessor (if it exists) is obtained based on its index position in the target text. This historical semantic information is used to determine the semantic state value of the third text unit itself. Each text unit in the target text is processed recursively in this way.

[0130] S1043. Multiply the key vector and value vector element by element to obtain the key-value interaction vector of the first text unit.

[0131] It should be noted that, as can be seen from step S102, the key vector and the value vector are obtained from the text unit vector through the key vector matrix and the value vector matrix with the same structure, respectively. That is, the key vector and the value vector have the same dimension, so element-wise multiplication between the vectors can be performed.

[0132] It should also be noted that the key vector and value vector each carry different aspects of information about the text unit. By multiplying these two types of information element by element, the resulting key-value interaction vector can comprehensively reflect the characteristics of the third text unit in different dimensions, thereby better understanding the semantic role of the text unit in the text and its semantic relationship with other text units.

[0133] For example, the key vector of the third text unit is k = [0.2, 0.5, 0.3], and the value vector is v = [0.4, 0.1, 0.6]. First, we multiply the elements of the key and value vectors of this third text unit along each dimension: 0.2 × 0.4 = 0.08, 0.5 × 0.1, 0.3 × 0.6 = 0.18. Then, we assemble the results of multiplying the elements along each dimension into a key-value interaction vector, kv = [0.08, 0.05, 0.18].

[0134] S1044. Based on the semantic state value of the third text unit, the key-value interaction vector of the first text unit, and the vector, determine the semantic state value of the first text unit.

[0135] In some embodiments, different weights can be assigned to the semantic state value of the third text unit and the key-value interaction vector of the first text unit to flexibly control the proportion of the semantic information of the text unit itself or the semantic information of the preceding text units in the semantic state value of the current text unit.

[0136] For example, the process of determining the semantic state value of the first text unit by assigning weights to the semantic state value of the third text unit and the key-value interaction vector of the first text unit can be represented as: s t =βs t-1 +(1-β)(k t v t )+h t Formula (5)

[0137] Among them, s t s represents the semantic state value corresponding to the first text unit. t-1 β represents the semantic state value corresponding to the third text unit, and k represents the weight of the semantic state value corresponding to the third text unit. t v t Let h represent the key-value interaction vector of the first text unit, 1-β represent the weight of the key-value interaction vector of the first text unit, and h represent the value of the key-value interaction vector of the first text unit. t This represents the text unit vector corresponding to the first text unit.

[0138] For example, suppose the vector representation of the first text unit is h. t = [0.1, 0.2, 0.3], the key-value interaction vector is kv = [0.08, 0.05, 0.18], and the semantic state value of the third text unit is s. t-1 = [0.2, 0.3, 0.4]. Let the weight β be 0.6, then 1-β is 0.4. The elements of each dimension in the semantic state value of the third text unit and the key-value interaction vector of the first text unit are weighted and summed, i.e.: βs t-1 [0]+(1-β)kt v t [0]=0.6×0.2+0.4×0.08=0.152, βs t-1 [1]+(1-β)k t v t [1]0.6×0.3+0.4×0.05=0.2, βs t-1 [2]+(1-β)k t v t [2] = 0.6 × 0.4 + 0.4 × 0.1 = 0.28. Finally, the weighted sum of each dimension element is added to the vector of the first text unit to obtain the semantic state value of the third text unit, i.e., [0.152, 0.2, 0.28] + [0.1, 0.2, 0.3] = [0.252, 0.4, 0.58].

[0139] The following section introduces how to obtain the feature representation of the first text unit.

[0140] In some embodiments, the attention representation and semantic state value of the first text unit may experience dimensionality mismatch when directly fused because the vectors used for computation have different dimensions. In this case, a linear projection fusion method can be used to align the dimensions of the two representations before computation to obtain the feature representation of the first text unit.

[0141] One possible implementation is to use a linear transformation matrix to linearly transform the attention representation corresponding to the first text unit, projecting its feature space onto the feature space of the semantic state value corresponding to the first text unit. In this case, the process is shown in Figure 5, and step S105 may further include the following steps:

[0142] S1051. Perform a linear transformation on the attention representation of the first text unit.

[0143] S1052. Add the linear transformation result of the attention representation of the first text unit to the semantic state value to determine the feature representation of the first text unit.

[0144] For example, the process of adding the linear transformation result of the attention representation of the first text unit to the semantic state value to determine the feature representation of each text unit can be represented as: y t =w o o t +s t Formula (6)

[0145] Among them, y t The feature representation of a text unit, w o This represents a linear transformation matrix.

[0146] For example, suppose the attention representation of the first text unit is o t = [1.2, -0.5, 0.8], semantic state value is s t = [0.4, 1.1, -0.3], and assume the linear transformation matrix is:

[0147] First, the attention representation is linearly transformed using a linear transformation matrix (row vectors multiplied by the matrix on the right):

[0148] That is, the element of each dimension in the attention representation is multiplied element-wise with the corresponding row of the matrix and then summed. The calculation process for the first dimension is: 1.2×0.5+(-0.5)×(-0.2)+0.8×0.1=0.78. Similarly, the calculation result for the second dimension is -0.26, and the calculation result for the third dimension is 0.34. The linear transformation result of the attention representation of this text unit is w. o o t = [0.78, -0.26, 0.34]. Next, based on formula (6), the linear transformation result of the attention representation of the text unit is added to the semantic state value to obtain the feature representation of the text unit, y. t =w o o t +s t = [0.78, -0.26, 0.34] + [0.4, 1.1, -0.3] = [1.18, 0.84, 0.04].

[0149] As can be seen from steps S1051-S1052, by mapping the attention representation to the same feature space as the semantic state value, the two can be fused under the same dimension and feature framework, so that the final feature representation can more comprehensively capture contextual dependencies and local key features, thereby improving the model's ability to understand classified text.

[0150] Another possible implementation is to use a linear transformation matrix to linearly transform the semantic state value corresponding to the first text unit, and project its feature space onto the feature space of the attention representation corresponding to the first text unit. This process can be implemented in the previous way, and will not be described in detail here.

[0151] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, the feature representation determination device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0152] In an exemplary embodiment, this application also provides a feature representation determination apparatus. This feature representation determination apparatus may include one or more functional modules for implementing the feature representation determination method of the above method embodiments.

[0153] For example, as shown in FIG6, the device for determining the feature representation includes an acquisition module 610 and a processing module 620.

[0154] The acquisition module 610 is used to acquire a first text unit, and to perform vectorization processing on the first text unit to acquire the vector corresponding to the first text unit. The first text unit is any text unit obtained by segmentation in the target text.

[0155] The processing module 620 is used to: determine, based on vectors, the key vector, value vector, and weight vector corresponding to the first text unit, whereby the weight vector represents the feature importance of the first text unit; determine the attention representation corresponding to the first text unit based on the key vector and value vector; determine the semantic state value corresponding to the first text unit based on the vectors, key vector, and value vector; and determine the feature representation of the first text unit based on the attention representation and semantic state value.

[0156] One possible implementation involves the processing module 620 specifically performing the following steps: Based on the index position of the first text unit in the target text, obtain at least one second text unit, where the index of each second text unit in the target text is less than or equal to the index of the first text unit in the target text. Obtain the weight vector and value vector corresponding to each second text unit. Normalize the weight vector corresponding to each second text unit to determine the attention weight of each second text unit. Based on the attention weight and value vector of each second text unit, determine the attention representation of the first text unit.

[0157] One possible implementation is that the processing module 620 is specifically used to: determine the attention weight of each second text unit according to the following formula:

[0158] Where, α i w represents the attention weight corresponding to the i-th second text unit. i Let represent the weight vector corresponding to the i-th second text unit, t represent the total number of second text units, and exp represent the exponential operation on the weight vector.

[0159] One possible implementation is that the processing module 620 is specifically used to: perform a weighted summation of the value vectors of each second text unit based on the attention weights of each second text unit to obtain the attention representation of the first text unit.

[0160] One possible implementation is that the processing module 620 is specifically used to: obtain the attention representation of the first text unit according to the following formula:

[0161] Among them, o t The attention representation of the first text unit, α i v represents the attention weight corresponding to the i-th second text unit. i Let represent the value vector corresponding to the i-th second text unit, and t represent the total number of second text units.

[0162] One possible implementation involves the processing module 620 specifically performing the following steps: Based on the index position of the first text unit in the target text, obtain the third text unit, where the third text unit is the preceding text unit of the first text unit in the target text. Obtain the semantic state value of the third text unit. Multiply the key vector and value vector element-wise to obtain the key-value interaction vector of the first text unit. Based on the semantic state value of the third text unit, the key-value interaction vector of the first text unit, and the vector, determine the semantic state value of the first text unit.

[0163] In one possible implementation, processing module 620 is specifically used to: determine the semantic state value of the third text unit according to the following formula: s t =βs t-1 +(1-β)(k t v t )+h t

[0164] Among them, s t s represents the semantic state value corresponding to the first text unit. t-1 β represents the semantic state value corresponding to the third text unit, and k represents the weight of the semantic state value corresponding to the third text unit. t v t Let h represent the key-value interaction vector of the first text unit, 1-β represent the weight of the key-value interaction vector of the first text unit, and h represent the value of the key-value interaction vector of the first text unit. tThis represents the text unit vector corresponding to the first text unit.

[0165] One possible implementation is that the processing module 620 is specifically used to: perform a linear transformation on the attention representation of the first text unit; add the result of the linear transformation of the attention representation of the first text unit to the semantic state value to determine the feature representation of the first text unit.

[0166] It should be noted that the module division in Figure 6 is illustrative and represents only one logical functional division. In actual implementation, other division methods are possible. For example, two or more functions can be integrated into a single processing module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0167] In exemplary embodiments, as described above, the computing device may specifically be an electronic device with computing processing capabilities, such as a computer or service. In this case, this application embodiment also provides an electronic device, and FIG7 is a schematic diagram of the composition of an electronic device provided in this application embodiment. As shown in FIG7, the electronic device includes: a processor 10, a memory 20, a communication line 30, a communication interface 40, and an input / output interface 50.

[0168] The processor 10, memory 20, communication interface 40, and input / output interface 50 can be connected via communication line 30.

[0169] Processor 10 is used to execute instructions stored in memory 20 to implement the feature representation determination method provided in the above embodiments of this application. Processor 10 may be a CPU, a network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller (MCU) / single-chip microcomputer / microcontroller, a programmable logic device (PLD), or any combination thereof. Processor 10 may also be any other device with processing functions, such as a circuit, device, or software module, which is not limited in this embodiment. In one example, processor 10 may include one or more CPUs, such as CPU0 and CPU1 in FIG. 7. As an optional implementation, the electronic device may include multiple processors, for example, in addition to processor 10, it may also include processor 60 (shown as an example by dashed lines in FIG. 7).

[0170] The memory 20 is used to store instructions. For example, the instructions may be computer programs. Optionally, the memory 20 may be a read-only memory (ROM) or other types of static storage devices that can store static information and / or instructions; it may also be a random access memory (RAM) or other types of dynamic storage devices that can store information and / or instructions; it may also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices, etc. The embodiments of this application do not limit this.

[0171] It should be noted that the memory 20 can exist independently of the processor 10 or it can be integrated with the processor 10. The memory 20 can be located inside or outside the electronic device, and this application embodiment does not impose any restrictions on this.

[0172] Communication line 30 is used to transmit information between the components included in the electronic device.

[0173] Communication interface 40 is used to communicate with other devices or other communication networks. These other communication networks can be Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. Communication interface 40 can be a module, circuit, transceiver, or any device capable of enabling communication.

[0174] Input / output interface 50 is used to enable human-computer interaction between users and electronic devices. For example, it enables action interaction or information exchange between users and electronic devices.

[0175] For example, the input / output interface 50 can be a mouse, keyboard, display screen, or touch screen. Action or information interaction between the user and the electronic device can be achieved through a mouse, keyboard, display screen, or touch screen.

[0176] It should be noted that the structure shown in Figure 7 does not constitute a limitation on the electronic device. In addition to the components shown in Figure 7, the electronic device may include more or fewer components than shown, or combinations of certain components, or different arrangements of components.

[0177] In an exemplary embodiment, this application also provides a readable storage medium including software instructions that, when run on an electronic device, cause the electronic device to perform any of the methods provided in the above embodiments.

[0178] In an exemplary embodiment, this application also provides a computer program product containing computer execution instructions, which, when run on an electronic device, causes the electronic device to perform any of the methods provided in the above embodiments.

[0179] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer-executable instructions. When these computer-executable instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer-executable instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer-executable instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks, SSDs).

[0180] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple components. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0181] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

[0182] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for determining a feature representation, the method comprising: Obtain the first text unit and perform vectorization processing on the first text unit to obtain the vector corresponding to the first text unit; the first text unit is any text unit obtained by segmenting the target text. Based on the vector, the key vector, value vector, and weight vector corresponding to the first text unit are determined; the weight vector is used to represent the feature importance of the first text unit. Based on the key vector and value vector corresponding to the first text unit, determine the attention representation corresponding to the first text unit; Based on the vector corresponding to the first text unit, the key vector, and the value vector, the semantic state value corresponding to the first text unit is determined; the semantic state value is used to represent the degree of semantic association between the first text unit and the text units preceding the first text unit. Based on the attention representation and semantic state value corresponding to the first text unit, the feature representation of the first text unit is determined.

2. The method according to claim 1, wherein, Determining the weight vector corresponding to the first text unit based on the vector includes: Obtain the weight vector matrix; the weight vector matrix is ​​used to convert a vector into a weight vector; Based on the vector and the weight vector matrix, the weight vector corresponding to the first text unit is determined.

3. The method according to claim 1, wherein, Based on the key vector and value vector corresponding to the first text unit, the attention representation corresponding to the first text unit is determined, including: Based on the index position of the first text unit in the target text, at least one second text unit is obtained; the index of each second text unit in the target text is less than or equal to the index of the first text unit in the target text. Obtain the weight vector and value vector corresponding to each second text unit; The weight vectors corresponding to each second text unit are normalized to determine the attention weights of each second text unit. The attention representation of the first text unit is determined based on the attention weights and value vectors of each second text unit.

4. The method according to claim 3, wherein, The step of normalizing the weight vector corresponding to each second text unit to determine the attention weight of each second text unit includes: The attention weights for each second text unit are determined using the following formula: Where, α i w represents the attention weight corresponding to the i-th second text unit; i Let represent the weight vector corresponding to the i-th second text unit; t represents the total number of second text units; exp represents the exponential operation on the weight vector.

5. The method according to claim 3, wherein, Determining the attention representation of the first text unit based on the attention weights and value vectors of each second text unit includes: Based on the attention weights of each second text unit, the value vectors of each second text unit are weighted and summed to obtain the attention representation of the first text unit.

6. The method according to claim 5, wherein, The step of weighted summing of the value vectors of each second text unit based on the attention weights of each second text unit to obtain the attention representation of the first text unit includes: The attention representation of the first text unit is obtained according to the following formula: Among them, o t The attention representation of the first text unit; α i v represents the attention weight corresponding to the i-th second text unit; i t represents the value vector corresponding to the i-th second text unit; t represents the total number of second text units.

7. The method according to claim 1, wherein, Determining the semantic state value corresponding to the first text unit based on the vector corresponding to the first text unit, the key vector, and the value vector includes: Based on the index position of the first text unit in the target text, the third text unit is obtained; the third text unit is the text unit preceding the first text unit in the target text. Obtain the semantic state value of the third text unit; The key vector and value vector are multiplied element by element to obtain the key-value interaction vector of the first text unit; The semantic state value of the first text unit is determined based on the semantic state value of the third text unit, the key-value interaction vector of the first text unit, and the vector.

8. The method according to claim 7, wherein, The step of determining the semantic state value of the first text unit based on the semantic state value of the third text unit, the key-value interaction vector of the first text unit, and the vector includes: The semantic state value of the third text unit is determined according to the following formula: s t =βs t-1 +(1-β)(k t v t )+h t ; Among them, s t This represents the semantic state value corresponding to the first text unit; s t-1 β represents the semantic state value corresponding to the third text unit; β represents the weight of the semantic state value corresponding to the third text unit; k t v t denoted as the key-value interaction vector of the first text unit; 1-β represents the weight of the key-value interaction vector of the first text unit; h t This represents the text unit vector corresponding to the first text unit.

9. The method according to claim 1, wherein, The step of determining the feature representation of the first text unit based on the attention representation and semantic state value corresponding to the first text unit includes: Perform a linear transformation on the attention representation of the first text unit; The linear transformation result of the attention representation of the first text unit is added to the semantic state value to determine the feature representation of the first text unit.

10. A feature representation determination device, the device comprising: Acquisition module, processing module; The acquisition module is used to acquire a first text unit and perform vectorization processing on the first text unit to acquire the vector corresponding to the first text unit; the first text unit is any text unit obtained by segmentation in the target text; The processing module is used to determine the key vector, value vector, and weight vector corresponding to the first text unit based on the vector; the weight vector is used to represent the feature importance of the first text unit. Based on the key vector and value vector corresponding to the first text unit, determine the attention representation corresponding to the first text unit; Based on the vector corresponding to the first text unit, the key vector, and the value vector, the semantic state value corresponding to the first text unit is determined; based on the attention representation and semantic state value corresponding to the first text unit, the feature representation of the first text unit is determined.

11. An electronic device, comprising: Processor and memory; The memory stores instructions that the processor can execute; When the processor is configured to execute the instructions, the electronic device performs the method as described in any one of claims 1-9.

12. A computer-readable storage medium, the computer-readable storage medium comprising: Computer software instructions; When the computer software instructions are executed in an electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1-9.