Text processing method and apparatus, and computer-readable storage medium

Abstract

The present disclosure relates to the technical field of computers, and in particular to a text processing method and apparatus, and a computer-readable storage medium. The text processing method comprises: for each self-attention layer among a plurality of self-attention layers of a self-attention network, acquiring a key channel of each self-attention layer; computing an estimation result of an attention weight for a plurality of objects of the self-attention network according to the key channel and target text; according to the estimation result, selecting, from among the plurality of objects, a plurality of target objects sharing the attention weight; and generating a processing result by sharing the attention weight among the plurality of target objects.

Description

Text processing methods, apparatus and computer-readable storage media Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a text processing method, apparatus and computer-readable storage medium. Background Technology

[0002] The Self Attention Mechanism is a core component of the Transformer architecture. It enhances the model's ability to pay attention to different aspects of the input data during inference, thereby capturing richer information features.

[0003] The basic idea of ​​attention mechanisms is to highlight the input parts that the model is interested in by assigning a weight to each element in the input sequence. In Transformer, a commonly used attention mechanism is "Scaled Dot-Product Attention". Scaled Dot-Product Attention is usually a dense attention mechanism, which requires calculating attention weights for every pair of positions in the entire input sequence.

[0004] Summary of the Invention

[0005] According to a first aspect of some embodiments of this disclosure, a text processing method is provided, comprising: for each of a plurality of attention layers of an attention network, obtaining a key channel of each attention layer; calculating an estimated result of attention weights for a plurality of objects of the attention network based on the key channels and target text; selecting a plurality of target objects that share attention weights from the plurality of objects based on the estimated result; and generating a processing result by sharing attention weights among the plurality of target objects.

[0006] According to a second aspect of some embodiments of this disclosure, a text processing apparatus is provided, comprising: an acquisition module configured to acquire a key channel for each of a plurality of attention layers of an attention network; a calculation module configured to calculate an estimated result of attention weights for a plurality of objects of the attention network based on the key channels and target text; a selection module configured to select a plurality of target objects sharing attention weights from the plurality of objects based on the estimated result; and a generation module configured to generate a processing result by sharing attention weights among the plurality of target objects.

[0007] According to a third aspect of some embodiments of the present disclosure, a text processing apparatus is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute a text processing method according to some embodiments of the present disclosure based on instructions stored in the memory.

[0008] According to a fourth aspect of some embodiments of the present disclosure, a computer-readable storage medium is provided having computer program instructions stored thereon, which, when executed by a processor, implement the text processing method according to some embodiments of the present disclosure.

[0009] According to a fifth aspect of some embodiments of the present disclosure, a computer program product is provided, including computer program instructions that, when executed by a processor, implement the text processing method described in some embodiments of the present disclosure.

[0010] This summary section is provided to briefly introduce the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0011] Other features and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0012] Preferred embodiments of the present disclosure are described below with reference to the accompanying drawings. The accompanying drawings, which are included to provide a further understanding of the present disclosure, and which, together with the following detailed description, are incorporated in and form a part of this specification and are used to explain the present disclosure. It should be understood that the drawings described below only relate to some embodiments of the present disclosure and are not intended to limit the present disclosure. In the drawings:

[0013] Figure 1 shows a schematic flowchart of a text processing method according to some embodiments of the present disclosure;

[0014] Figure 2 illustrates a flowchart of obtaining key channels according to some embodiments of this disclosure;

[0015] Figure 3 illustrates a schematic diagram of attention vectors in some embodiments of this disclosure;

[0016] Figure 4 illustrates a flowchart of the estimated attention weights according to some embodiments of this disclosure;

[0017] Figure 5 illustrates a schematic diagram of lexical shared attention according to some embodiments of the present disclosure;

[0018] Figure 6 illustrates a schematic diagram of shared attention according to some embodiments of the present disclosure;

[0019] Figure 7 shows a block diagram of a text processing apparatus according to some embodiments of the present disclosure;

[0020] Figure 8 shows a block diagram of a text processing apparatus according to other embodiments of the present disclosure;

[0021] Figure 9 shows a block diagram of an electronic device according to some embodiments of the present disclosure.

[0022] It should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not necessarily drawn to actual scale. The same or similar reference numerals are used in the various drawings to denote the same or similar parts. Therefore, once an item is defined in one drawing, it may not be discussed further in subsequent drawings. Detailed Implementation

[0023] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. However, it is obvious that the described embodiments are only some embodiments of this disclosure, and not all embodiments. The following description of the embodiments is merely illustrative and is in no way intended to limit this disclosure or its application or use. It should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein.

[0024] It should be understood that the various steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect. Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of components and steps set forth in these embodiments should be interpreted as merely exemplary and do not limit the scope of this disclosure.

[0025] As used in this disclosure, the term "comprising" and its variations are open-ended terms that include at least the following elements / features but do not exclude other elements / features, i.e., "including but not limited to". Furthermore, as used in this disclosure, the term "including" and its variations are open-ended terms that include at least the following elements / features but do not exclude other elements / features, i.e., "including but not limited to". Therefore, "comprising" and "including" are synonymous. The term "based on" means "at least partially based on".

[0026] Throughout this specification, the terms "one embodiment," "some embodiments," or "embodiment" mean that a specific feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of the invention. For example, the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; and the term "some embodiments" means "at least some embodiments." Furthermore, the appearance of the phrases "in one embodiment," "in some embodiments," or "in an embodiment" in various places throughout the specification does not necessarily refer to the same embodiment, but may refer to the same embodiment.

[0027] It should be noted that the concepts of "first," "second," etc., used in this disclosure are used only to distinguish different devices, modules, or units, and are not intended to define the order of functions performed by these devices, modules, or units or their interdependencies. Unless otherwise specified, the concepts of "first," "second," etc., are not intended to imply that the objects described herein must be in a given temporal, spatial, rank, or any other given order.

[0028] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0029] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0030] The embodiments of this disclosure are described in detail below with reference to the accompanying drawings; however, this disclosure is not limited to these specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. Furthermore, in one or more embodiments, specific features, structures, or characteristics can be combined in any suitable manner that will be apparent to those skilled in the art from this disclosure.

[0031] In related technologies, dense attention mechanisms require calculating attention weights for every pair of positions in the entire input sequence. This means that for each token generated during inference, the attention needs to be recalculated. Furthermore, due to the deep layers in current large models, the attention mechanism of each layer also needs to be calculated independently. Therefore, the computational overhead of these attention weights during inference is significant, limiting the speed at which the model processes text.

[0032] This disclosure provides a text processing method, apparatus, and computer-readable storage medium that can improve the speed of text processing while ensuring a certain level of accuracy.

[0033] Figure 1 shows a schematic flowchart of a text processing method according to some embodiments of the present disclosure.

[0034] As shown in Figure 1, the text processing method includes: Step S1, for each of the multiple attention layers of the attention network, obtaining the key channel of each attention layer; Step S2, based on the key channel and the target text, calculating the estimated result of the attention weight for multiple objects of the attention network; Step S3, based on the estimated result, selecting multiple target objects that share the attention weight from the multiple objects; Step S4, generating the processing result by sharing the attention weight among the multiple target objects.

[0035] The key channel of an attention layer is the key channel of the attention vector in that layer. For example, the dimension of the attention vector is d. h In other words, the attention vector has d h There are 1 channel, and the key channel is, for example, at least one important channel, d. h The value is a positive integer. Attention vectors can be different across different attention layers, and key channels can also be different. For example, attention vectors can be the query vector Q and the key vector K.

[0036] The target text is, for example, the user's input text, and the processing result is feedback on the input text. For example, the user inputs a question, and the processing result generates the answer to the question.

[0037] Multiple objects in an attention network can be considered as processing objects of the network. For example, from the perspective of the overall network structure, the processing object of an attention layer is the input features of the current layer. From the perspective of words in the target text, each word can also be regarded as an object.

[0038] According to this disclosure, attention weights are first quickly estimated based on key channels. Then, based on the estimated attention weights, it is determined which elements can share attention weights, thereby accelerating text processing. For example, if two target objects share attention weights, the original requirement of calculating attention weights twice can be reduced to calculating them only once.

[0039] The text processing method in this embodiment can be executed on the client side or partially on the server side.

[0040] The following describes, with reference to Figure 2, a method for obtaining key channels according to some embodiments of the present disclosure.

[0041] As shown in Figure 2, step S1 includes: step S11, for each attention layer, determining the second attention vector of multiple sample texts; step S12, determining the key channel of each attention layer based on the second attention vector of multiple sample texts.

[0042] For example, key channels can be identified offline. Multiple sample texts can be obtained from an offline calibration dataset. The offline calibration dataset is used to statistically analyze the value distribution of the attention network along the feature channel dimension. Based on the value distribution of the feature channel dimension of each attention layer, the key channels of that layer, i.e., the key points of the channels, are determined.

[0043] Typically, the activation values ​​of neural networks need to be calculated online in real time. However, selecting key channels online not only fails to accelerate inference but may also incur additional computational overhead. Therefore, an offline method can be used to identify the k channels that have a significant impact on model performance and select these locations as key channels. For key channels, their channel indices can be recorded.

[0044] The attention vector includes multiple channels. Based on the second attention vector of multiple sample texts, the key channels of each attention layer are determined, including: for each attention layer, the key channels of each attention layer are determined based on the mean of the components of the second attention vector of multiple sample texts in each channel.

[0045] Key channels can be selected in several ways: select k key channels for Q; select k key channels for K; select k key channels for Q*K, where k is a positive integer. These methods can be used individually or in combination.

[0046] Taking the selection of k key channels for Q and / or K as an example, the second attention vector is the query-key pair vector used when selecting key channels offline, such as including the second query vector Q and the second key vector K, both with a feature dimension of d. h When using an offline calibration dataset, Q and K can be obtained for each sample at layer l. By averaging Q and K across the sample dimension for each layer, the average value of Q and K across the feature channels can be obtained for each layer in the attention network. Based on the average value, the magnitude of each channel's influence on the network can be determined.

[0047] At least one channel whose absolute value of the component mean is ranked first can be identified as the key channel for each attention layer. For example, for each layer, the top k channels with the largest absolute value of the mean in the second attention vector can be selected as key channels.

[0048] Alternatively, k key channels can be selected for Q*K. For example, for each attention layer, the key channels of each attention layer can be determined by the average of the products of the second query vector and the second key vector of multiple sample texts in each channel.

[0049] For example, for the same network layer, the Q*K calculated from multiple samples is averaged in the sample dimension to obtain the distribution of Q*K in the channel dimension. Based on the distribution of Q*K in the channel dimension, the key channels of each attention layer are determined.

[0050] At least one channel whose absolute value of the mean of the product of the components is ranked first can be identified as the key channel for each attention layer. For example, the top k channels with the largest absolute value of the mean of the product of the second attention vectors can be used as key channels.

[0051] Each sample has multiple words. For each attention layer, the key channel of each attention layer is determined based on the mean of the components of the second attention vector of multiple words in each channel of the multiple sample texts.

[0052] For example, for each sample, first calculate Q, K, and Q*K for each word element. Then, average Q, K, and Q*K for all words elements across all samples to obtain the channel-dimension distribution of Q, K, and Q*K for all samples.

[0053] The resulting distribution is shown in Figure 3. In the sense of the average of the calibration dataset, the channels containing the k largest average values ​​are selected as channels. That is, the channels with the largest absolute values ​​of Q, K, and / or Q*K can be selected as key channels, and the indices m of these key channels are recorded.

[0054] In step S1, the key channels have been acquired offline. Therefore, the process of estimating attention weights can then be performed online and can be processed according to the target text. The following, with reference to Figure 4, describes a method for estimating attention weights according to some embodiments of this disclosure.

[0055] As shown in Figure 4, step S2 includes: step S21, determining the first attention vector of the target text based on the target text; step S22, determining the channels involved in the prediction in the first attention vector based on the key channels; and step S23, calculating the prediction result of the attention weight based on the channels involved in the prediction.

[0056] For example, when estimating attention weights, only the key channels among the multiple channels of the first attention vector participate in the calculation. Compared to having all channels participate in the calculation, having only some channels participate can improve the estimation speed and reduce the amount of computation.

[0057] The first attention vector is a query-key pair vector used in online prediction of attention weights. For example, it includes a first query vector Q and a first key vector K. The product of the corresponding channel components involved in the prediction in the first query vector and the first key vector can be determined first. The prediction result of the attention weight is obtained based on the sum of the products of the channel components involved in the prediction.

[0058] For example, if channel 2 is the critical channel, then the component of the Q vector in channel 2 is multiplied by the component of the K vector in channel 2 to obtain the product. The products of the components of all channels involved in the prediction are summed to obtain the prediction result. Channels not involved in the prediction do not require product calculation or summation.

[0059] The method for quickly estimating attention weights can be expressed as follows.

[0060] Where Q' and K' represent attention matrices with keypoint masks, m represents key channel indices, and M represents the set of key channel indices.

[0061] Compared to the original requirement of d h This disclosure only selects k key channels for multiplication and addition when estimating attention weights, instead of the usual vector multiplication and addition. In other words, it only requires a fraction of the original time... This allows for rapid prediction of attention weights. (Using d) h Taking 128 as an example, if 16 channels are selected as keypoints, the predicted attention weights can be obtained in only 1 / 8 of the time. In other words, the method for predicting attention weights based on key channels disclosed in this paper can reduce computational overhead and the number of multiplication operations.

[0062] After estimating the attention weights, we can determine which objects can share the attention weights, which will be explained in detail below.

[0063] Multiple objects include, for example, an adjacent first word and a second word, with the second word following the first word. Based on the prediction results, multiple target objects that share attention weights are selected from the multiple objects, including: determining whether the first word and the second word share attention weights based on the prediction results of the attention weight of the first word and the prediction results of the attention weight of the second word relative to the word preceding the second word.

[0064] For example, for each word, its attention weight relative to all current word elements is estimated. Then, based on the estimated weights, it is further determined whether adjacent word elements can share attention weights. If a group of adjacent word elements can share attention weights, it is identified as the target.

[0065] In some embodiments, in response to the similarity between the predicted attention weight of the first word and the predicted attention weight of the second word relative to the word preceding the second word exceeding a first threshold, it is determined that the first word and the second word share attention weights.

[0066] For example, attention weights are shared only when the estimated attention weights of adjacent words are similar, thus avoiding a severe drop in accuracy caused by sharing.

[0067] Two adjacent lexical units are denoted as q i and q i-1 Quickly predict q using key channels i and q i-1 The attention weight at a certain layer of the network is denoted as A. i and A i-1 , where A i The length of A is i. i-1 The length of A is i-1. i [-1] This term is q i A's attention to itself i-1 This item is not included. In other words, the predicted attention weight for the first word is represented as A. i-1 The predicted attention weight of the second word relative to the words preceding it is A. i [:i-1]. Similarity includes, for example, cosine similarity, calculated as A. i-1 and A i The cosine similarity between [:i-1] is as follows.

[0068] Similarly, A can also be calculated. i-1 and A i The Euclidean norm (L2 norm) of [:i-1] ||A i [:i-1]||,||A i-1 ||.

[0069] When the cosine similarity is greater than or equal to a threshold (e.g., 0.98) and / or A i-1 With A i When the L2 norm of [:i-1] is close (e.g., the ratio is in the interval [0.95, 1.05]), it is assumed that q i and q i-1 The attention weights are similar and can be shared. Taking the condition that both cosine similarity and norm must be satisfied as an example, the expression for determining whether they can be shared is as follows.

[0070] Under the condition of satisfying formula (3), it is determined that these two adjacent dimensions can share attention weights in the current network layer.

[0071] In some embodiments, multiple objects include a first word and a second word sharing attention weights, the second word being adjacent to the first word and following the first word, and a processing result is generated by sharing attention weights among multiple target objects, including: calculating the actual result of the attention weight of the first word; determining the actual result of the attention weight of the first word as the actual result of the attention weight of the second word relative to the word preceding the second word; and generating a processing result based on the actual result of the attention weight of the first word and the actual result of the attention weight of the second word relative to the word preceding the second word.

[0072] For example, as shown in Figure 5, q is determined through prediction. i and q i-1 If attention weights can be shared, then when calculating q... i-1 After the actual value of the attention weights (i.e., the accurate attention weights), q i Relative to q i-1 and q i-1 The attention weights of the previous lexical units can be directly used with q. i-1 The actual values ​​of the attention weights do not need to be calculated again.

[0073] The formula for calculating the actual value of attention weights is expressed as follows.

[0074] In some embodiments, generating a processing result based on the actual result of the attention weight of the first word element and the actual result of the attention weight of the second word element relative to the word elements preceding the second word element includes: calculating the actual result of the attention weight of the second word element relative to itself; determining the actual result of the attention weight of the second word element based on the actual result of the attention weight of the second word element relative to the word elements preceding the second word element and the actual result of the attention weight of the second word element relative to itself; and generating a processing result based on the actual result of the attention weight of the first word element and the actual result of the attention weight of the second word element.

[0075] As shown in Figure 5, even with attention sharing, A i [-1] This item still needs to be calculated separately. Let A... i-1 With the calculated A i [-1] Only by splicing together can we obtain A. i .

[0076] In some embodiments, the plurality of objects include adjacent first attention layers and second attention layers, with the second attention layer following the first attention layer. Based on the prediction results, the second attention layer selects a plurality of target objects that share attention weights from the plurality of objects, including: determining whether the first attention layer and the second attention layer share attention weights based on the prediction results of the attention weights of the first attention layer and the prediction results of the attention weights of the second attention layer.

[0077] Similar to determining whether attention can be shared between lexical units, for each attention layer in the network, the attention weight of the object it processes is estimated. Then, based on the estimated weights, it is further determined whether adjacent attention layers can share attention weights. If they can share attention weights, then this group of adjacent attention layers is the target object.

[0078] In some embodiments, determining whether the first attention layer and the second attention layer share attention weights based on the predicted attention weights of the first attention layer and the predicted attention weights of the second attention layer includes: determining whether the first attention layer and the second attention layer share attention weights in response to the similarity between the predicted attention weights of the first attention layer and the predicted attention weights of the second attention layer exceeding a second threshold.

[0079] Similar to determining whether attention can be shared between lexical units, we can determine whether two adjacent attention layers can share attention based on the estimated similarity between their attention weights (e.g., cosine similarity, L2 norm, etc.).

[0080] In some embodiments, multiple objects include a first attention layer and a second attention layer that share attention weights. The second attention layer is adjacent to the first attention layer and is located after the first attention layer. By sharing attention weights among multiple target objects, a processing result is generated, including: calculating the actual result of the attention weights of the first attention layer; determining the actual result of the attention weights of the first attention layer as the actual result of the attention weights of the second attention layer; and generating a processing result based on the actual result of the attention weights of the first attention layer and the actual result of the attention weights of the second attention layer.

[0081] For example, if the first attention layer and the second attention layer have been determined to share attention weights, there is no need to calculate the actual result of the attention weights of the second attention layer after calculating the actual result of the attention weights of the first attention layer.

[0082] Unlike the sharing of attention between lexical units, the shape of the attention weights between different layers is the same. Therefore, the second attention layer can directly use the actual result of the attention weights of the first attention layer as the attention weights of this layer, without having to calculate a separate set of attention weights.

[0083] Attention weights can be shared only between lexical units or only between attention layers, or they can be shared both between lexical units and between attention layers.

[0084] The attention-sharing patterns of some embodiments of this disclosure are described below with reference to Figure 6.

[0085] As can be seen from Figure 6, attention can be shared from two dimensions: lexical units and network layers. These are two mutually perpendicular sharing lines, meaning that these two sharing modes can simultaneously accelerate inference.

[0086] Unlike large-model sparse algorithms in related technologies that discard some channels or words when calculating attention weights, some embodiments of this disclosure accelerate network inference computation by sharing attention weights between words and / or between neural network layers. Furthermore, this disclosure determines whether to share attention weights by predicting their differences, achieving inference acceleration while maintaining a certain level of accuracy, and avoiding a significant drop in accuracy caused by sharing attention weights when there are large differences in their values.

[0087] According to this disclosure, whenever the input word q is used for inference, a cache is needed to store the estimated attention weights and actual attention weights of each layer of the network. In the multi-head attention mechanism, both vectors have a shape and size of [head_num, seq_len], and the number of layers in the network is num_layer. Assuming that a value requires approximately 8 bytes of storage space, the additional storage size required is num_layer * head_num * seq_len * 2 * 8 bytes. Taking num_layer and head_num as both 32 as an example, and assuming an average seq_len of 1k, the additional storage size required is 2048000 * 8 bytes, which is approximately 15 megabytes. This is a completely acceptable additional storage size.

[0088] Since a similarity threshold is required for attention weights to be shared between different words and layers, the computational savings can only be estimated on an average basis. Assuming that attention weights can be shared between every two words and every two layers (in reality, multiple consecutive words or multiple consecutive layers may share the same attention weight matrix), then compared to before sharing, only an average of (1-1 / 2)*(1-1 / 2) = 1 / 4 of the attention weights need to be calculated. The additional computational cost for predicting attention is negligible; as analyzed earlier, it only requires an additional 1 / 8 of the computation. Therefore, the key-channel-based attention-sharing mechanism proposed in this disclosure can improve the inference speed of large models by approximately 4 times.

[0089] Figure 7 shows a block diagram of a text processing apparatus according to some embodiments of the present disclosure.

[0090] As shown in Figure 7, the text processing device 7 includes: an acquisition module 71 configured to acquire the key channel of each attention layer of the multiple attention layers of the attention network; a calculation module 72 configured to calculate the estimated result of attention weights for multiple objects of the attention network based on the key channel and the target text; a selection module 73 configured to select multiple target objects that share attention weights from the multiple objects based on the estimated result; and a generation module 74 configured to generate a processing result by sharing attention weights among the multiple target objects.

[0091] The acquisition module 71 of the text processing device 7 can be used to execute step S1 of FIG1. ​​The calculation module 72 of the text processing device 7 can be used to execute step S2 of FIG1. ​​The selection module 73 of the text processing device 7 can be used to execute step S3 of FIG1. ​​The generation module 74 of the text processing device 7 can be used to execute step S4 of FIG1.

[0092] Figure 8 shows a block diagram of a text processing apparatus according to other embodiments of the present disclosure.

[0093] As shown in FIG8, the text processing device 8 includes: a memory 81; and a processor 82 coupled to the memory 81, the processor 82 being configured to execute the text processing method of any of the foregoing embodiments based on instructions stored in the memory 81.

[0094] Memory 81 is used to store one or more computer-readable instructions. Memory 81 may include any combination of various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory, including but not limited to random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), and flash memory. Memory 81 may, for example, store operating systems, application programs, boot loaders, databases, and other programs, as well as various application programs and various data.

[0095] The processor 82 is configured to execute computer-readable instructions to implement the text processing method of any of the foregoing embodiments. Specific implementation details of each step of the text processing method can be found in the above embodiments; repeated details will not be elaborated upon here.

[0096] Processor 82 can be configured to execute steps S1-S4 of Figure 1. Processor 82 can be various processing devices, such as a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The central processing unit (CPU) can be an x86 or ARM architecture, etc.

[0097] The processor 82 and the memory 81 can communicate with each other directly or indirectly. For example, the processor 82 and the memory 81 can communicate via a network. The network can include wireless networks, wired networks, and / or any combination of wireless and wired networks. The processor 82 and the memory 81 can also communicate with each other via a system bus, which is not limited in this disclosure.

[0098] It should be noted that the components of the text processing device 8 shown in Figure 8 are merely exemplary and not limiting. The text processing device 8 may have other components depending on the specific application requirements. The processor 82 can control other components in the text processing device 8 to perform desired functions.

[0099] Text processing devices can be implemented by software, firmware, and / or hardware, and can be integrated into electronic devices with relevant applications installed.

[0100] Figure 9 shows a block diagram of an electronic device according to some embodiments of the present disclosure.

[0101] The electronic device 9 shown in Figure 9 can be a computer system with a dedicated hardware structure, capable of performing corresponding functions when relevant applications are installed.

[0102] Electronic devices include, but are not limited to, mobile terminals such as smartphones, laptops, personal digital assistants (PDAs), tablet computers (PCs), PMPs (portable multimedia players), in-vehicle terminals (such as in-vehicle navigation terminals), wearable devices, and fixed terminals such as digital televisions and desktop computers.

[0103] As shown in Figure 9, the Central Processing Unit (CPU) 91 executes various processes based on the program stored in the Read-Only Memory (ROM) 92 or the program loaded from the storage section 98 into the Random Access Memory (RAM) 93. The RAM 93 stores data required as needed when the CPU 91 executes various processes. The CPU is merely exemplary and can also be other types of processors, such as the various processors described above. The ROM 92, RAM 93, and storage section 98 can be various forms of computer-readable storage media. It should be noted that although the ROM 92, RAM 93, and storage section 98 are shown separately in Figure 8, one or more of them can be combined or located in the same or different memories or storage modules.

[0104] CPU 91, ROM 92 and RAM 93 are interconnected via bus 94. Input / output interface 95 is also connected to bus 94.

[0105] The following components are connected to the input / output interface 95: input section 96, such as a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output section 97, including displays such as cathode ray tube (CRT), liquid crystal display (LCD), speakers, vibrators, etc.; storage section 98, including hard disk, magnetic tape, etc.; and communication section 99, including network interface cards such as LAN cards, modems, etc. The communication section 99 allows communication processing to be performed via a network such as the Internet. It is readily understood that although the various devices or modules in the electronic device 9 shown in Figure 9 communicate via bus 94, they can also communicate via a network or other means, wherein the network can include wireless networks, wired networks, and / or any combination of wireless and wired networks.

[0106] As needed, drive 910 is also connected to input / output interface 95. Removable media 911, such as disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on drive 910 as needed, so that computer programs read from them can be installed into storage section 98 as needed.

[0107] When the above series of processes are implemented through software, the program constituting the software can be installed from a network such as the Internet or a storage medium such as a removable medium 911.

[0108] According to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product that, when run on a computer, causes the computer to implement the text processing methods of any of the foregoing embodiments. The computer program product includes a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 99, or installed from a storage portion 98, or installed from a ROM 92. When the computer program is executed by the CPU 91, the text processing methods of the embodiments of this disclosure are performed.

[0109] It should be noted that, in the context of this disclosure, a computer-readable medium can be a tangible medium that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0110] A computer-readable medium may be a computer-readable storage medium, a computer-readable signal medium, or any combination thereof.

[0111] Computer-readable storage media include, but are not limited to, systems, apparatuses, or devices that are electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. A computer program is stored on the computer-readable storage medium that, when executed by a processor, implements the text processing method of any of the foregoing embodiments.

[0112] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0113] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0114] In some embodiments, a computer program is also provided, comprising: instructions that, when executed by a processor, cause the processor to perform the text processing method of any of the above embodiments. For example, the instructions may be embodied in computer program code.

[0115] In embodiments of this disclosure, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0116] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0117] The functions described above can be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that can be used include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0118] While specific embodiments of this disclosure have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. A text processing method, comprising: For each of the multiple attention layers in an attention network, obtain the key channels of each attention layer; Based on the key channels and target text, the estimated results of attention weights are calculated for multiple objects in the attention network; Based on the prediction results, select multiple target objects that share attention weights from the multiple objects; The processing result is generated by sharing attention weights among the multiple target objects.

2. The text processing method according to claim 1, wherein, The step of calculating the estimated attention weights for multiple objects in the attention network based on the key channels and target text includes: Based on the target text, determine the first attention vector of the target text; Based on the key channels, determine the channels in the first attention vector that participate in the prediction; Based on the channels involved in the prediction, the prediction results of the attention weights are calculated.

3. The text processing method according to claim 2, wherein, The first attention vector includes a first query vector and a first key vector. The step of calculating the prediction result of the attention weights based on the channels involved in the prediction includes: Determine the product of the corresponding channel components involved in the prediction in the first query vector and the first key vector; The prediction result of the attention weight is obtained by summing the products of the components of the channels involved in the prediction.

4. The text processing method according to any one of claims 1-3, wherein, The plurality of objects includes adjacent first and second word units, wherein the second word unit follows the first word unit. The step of selecting multiple target objects sharing attention weights from the plurality of objects based on the prediction result includes: Based on the predicted attention weight of the first word and the predicted attention weight of the second word relative to the word preceding the second word, it is determined whether the first word and the second word share the attention weight.

5. The text processing method according to claim 4, wherein, The step of determining whether the first word and the second word share the attention weight based on the estimated attention weight of the first word and the estimated attention weight of the second word relative to the words preceding the second word includes: If the similarity between the predicted attention weight of the first word and the predicted attention weight of the second word relative to the preceding words exceeds a first threshold, then the first word is determined to be... The attention weights are shared between the lexical unit and the second lexical unit.

6. The text processing method according to any one of claims 1-5, wherein, The plurality of objects includes adjacent first attention layers and second attention layers, wherein the second attention layer follows the first attention layer. The step of selecting multiple target objects sharing attention weights from the plurality of objects based on the estimation result includes: Based on the predicted attention weights of the first attention layer and the predicted attention weights of the second attention layer, it is determined whether the first attention layer and the second attention layer share attention weights.

7. The text processing method according to claim 6, wherein, The step of determining whether the first attention layer and the second attention layer share attention weights based on the predicted attention weights of the first attention layer and the predicted attention weights of the second attention layer includes: If the similarity between the predicted attention weights of the first attention layer and the predicted attention weights of the second attention layer exceeds a second threshold, it is determined that the first attention layer and the second attention layer share attention weights.

8. The text processing method according to any one of claims 1-7, wherein, The plurality of objects includes a first word and a second word that share attention weights. The second word is adjacent to the first word and follows the first word. By sharing attention weights among the plurality of target objects, a processing result is generated, including: The actual result of calculating the attention weight of the first word element; The actual result of the attention weight of the first word element is determined as the actual result of the attention weight of the second word element relative to the word elements preceding the second word element; The processing result is generated based on the actual results of the attention weight of the first word and the actual results of the attention weight of the second word relative to the words preceding the second word.

9. The text processing method according to claim 8, wherein, The step of generating the processing result based on the actual result of the attention weight of the first lexical unit and the actual result of the attention weight of the second lexical unit relative to the lexical units preceding the second lexical unit includes: Calculate the actual result of the attention weight of the second word relative to itself; The actual result of the attention weight of the second word is determined based on the actual result of the attention weight of the second word relative to the word preceding the second word and the actual result of the attention weight of the second word relative to itself. The processing result is generated based on the actual results of the attention weights of the first word and the second word.

10. The text processing method according to any one of claims 1-9, wherein, The plurality of objects includes a first attention layer and a second attention layer that share attention weights. The second attention layer is adjacent to the first attention layer and follows the first attention layer. The step of generating a processing result by sharing attention weights among the plurality of target objects includes: The actual result of calculating the attention weights of the first attention layer; The actual result of the attention weights of the first attention layer is determined as the actual result of the attention weights of the second attention layer; The processing result is generated based on the actual results of the attention weights of the first attention layer and the actual results of the attention weights of the second attention layer.

11. The text processing method according to any one of claims 1-10, wherein, The acquisition of the key channels for each attention layer includes: For each attention layer, a second attention vector for multiple sample texts is determined; Based on the second attention vector of the plurality of sample texts, the key channel of each attention layer is determined.

12. The text processing method according to claim 11, wherein, The attention vector includes multiple channels, and determining the key channel of each attention layer based on the second attention vector of the multiple sample texts includes: For each attention layer, the key channel of each attention layer is determined based on the mean of the components of the second attention vector of the plurality of sample texts in each channel.

13. The text processing method according to claim 12, wherein, For each attention layer, the key channel of each attention layer is determined based on the mean of the components of the second attention vector of the plurality of sample texts in each channel, including: The channel whose absolute value of the mean of the components is ranked first is identified as the key channel of each attention layer.

14. The text processing method according to any one of claims 11-13, wherein, The second attention vector includes a second query vector and a second key vector. The step of determining the key channel of each attention layer based on the second attention vectors of the plurality of sample texts includes: For each attention layer, the key channel of each attention layer is determined based on the average of the products of the second query vector and the second key vector of the plurality of sample texts in each channel.

15. The text processing method according to claim 14, wherein, For each attention layer, the key channel of each attention layer is determined based on the average of the product of the second query vector of the plurality of sample texts and the components of the second key vector in each channel, including: The channel whose absolute value of the mean of the product of the components is ranked first is identified as the key channel of each attention layer.

16. The text processing method according to any one of claims 11-15, wherein, The step of determining the key channel of each attention layer based on the second attention vector of the plurality of sample texts includes: For each attention layer, the key channel of each attention layer is determined based on the mean of the components of the second attention vector of multiple lexical units of the multiple sample texts in each channel.

17. A text processing apparatus, comprising: The acquisition module is configured to acquire the key channel of each of the multiple attention layers of the attention network; The calculation module is configured to calculate the estimated attention weights for multiple objects of the attention network based on the key channel and the target text. The selection module is configured to select multiple target objects that share attention weights from the multiple objects based on the estimation results. The generation module is configured to generate processing results by sharing attention weights among the plurality of target objects.

18. A text processing apparatus, comprising: Memory; as well as A processor coupled to the memory, the processor being configured to execute the text processing method according to any one of claims 1 to 16 based on instructions stored in the memory.

19. A computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the text processing method according to any one of claims 1 to 16.

20. A computer program product comprising computer program instructions that, when executed by a processor, implement the text processing method according to any one of claims 1 to 16.

Images