Text segmentation method and apparatus, electronic device, and computer-readable storage medium
Patent Information
- Application Number
- PCT/CN2025/095833
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-02-25
- Filing Date
- 2025-05-19
- Publication Date
- 2026-09-03
Smart Images
Figure CN2025095833_03092026_PF_FP_ABST
Abstract
Description
Text segmentation methods, apparatuses, electronic devices, and computer-readable storage media
[0001] Cross-reference of related applications
[0002] This disclosure claims priority to Chinese Patent Application No. 202510210103.9, filed on February 25, 2025, entitled “Text Segmentation Method, Apparatus, Electronic Device and Computer-Readable Storage Medium”, the entire contents of which are incorporated herein by reference. Technical Field
[0003] This disclosure relates to the field of computer technology, and more specifically, to a text segmentation method, apparatus, electronic device, and computer-readable storage medium. Background Technology
[0004] With the groundbreaking progress of large models in the field of natural language processing, question answering systems based on large models have overcome the shortcomings of traditional question answering systems, such as their inability to effectively handle diverse user questions and mechanical responses. Because they can understand more complex language structures and capture richer contextual information, they can provide users with more accurate, fluent and personalized answers, and interact with users in a more human and intelligent way.
[0005] While large models improve the response quality of question-answering systems, their limited coverage of specific domain knowledge during training can lead to generated answers that, while seemingly reasonable and logically sound, are actually inaccurate or contradict reality. To address this, Retrieval-Augmented Generation (RAG) combines external knowledge retrieval with generative large models, providing robust support for large model-based question-answering systems. RAG enhances the knowledge coverage of large models by first retrieving relevant information from the database before generating answers, using the retrieved knowledge as contextual input to the model, thereby improving its accuracy and reliability.
[0006] The information in the database usually comes from external documents. The external documents need to be segmented into multiple fragments, and then each fragment is converted into a vector and stored in the database. Whether the external documents are segmented reasonably or not will affect the completeness of the context information retrieved by RAG, and thus affect the accuracy and reliability of the question-and-answer output of RAG. Summary of the Invention
[0007] One of the objectives of this disclosure is to provide a text segmentation method, apparatus, electronic device, and computer-readable storage medium that can reasonably segment text according to semantics, ensure the semantic coherence of the segmented semantic segments, and thereby improve the accuracy and reliability of RAG feedback question-and-answer output.
[0008] To achieve the above objectives, the technical solutions adopted in the embodiments of this disclosure are as follows:
[0009] Firstly, this disclosure provides a text segmentation method, the method comprising:
[0010] Obtain multiple sentences obtained after sentence-level segmentation of the target text;
[0011] Starting with the first sentence and ending with the last sentence among the multiple sentences, a preset sliding window is slid multiple times. After each slide, the multiple target sentences within the preset sliding window are semantically segmented to obtain at least one semantic segment.
[0012] For each semantic segment obtained from the segmentation, the semantic segments obtained from the current segmentation and the semantic segments obtained from the previous segmentation are clustered to obtain the target semantic segments to which each target sentence belongs within the preset sliding window after the previous sliding and after the current sliding, until the semantic segments to which each sentence belongs among the multiple sentences are obtained.
[0013] In an optional implementation, the step of semantically segmenting multiple target sentences within the preset sliding window to obtain at least one semantic segment includes:
[0014] The multiple target sentences are input into a pre-trained text segmentation model to obtain the semantic continuity of sentence pairs composed of two adjacent target sentences;
[0015] For any pair of target sentences, if the semantic continuity of the pair of target sentences is greater than a preset value, then the two target sentences in the pair of target sentences are assigned to the same semantic segment; otherwise, the two target sentences in the pair of target sentences are assigned to different semantic segments.
[0016] Each sentence pair is sequentially used as the target sentence pair to obtain at least one semantic segment after semantic segmentation of the multiple target sentences.
[0017] In an optional implementation, the first target sentence in the preset sliding window after this slide is the same as the last target sentence in the preset sliding window after the previous slide, and the two adjacent target sentences in the preset sliding window form a sentence pair;
[0018] The step of clustering the semantic segments obtained from each segmentation with the semantic segments obtained from the previous segmentation to obtain the target semantic segment to which each target sentence in the preset sliding window belongs after the previous sliding and the current sliding includes:
[0019] Obtain the semantic continuity of the first sentence pair within the preset sliding window after this sliding action;
[0020] Calculate the semantic similarity between the two target sentences in the first sentence pair;
[0021] The semantic association score of the first sentence pair is calculated based on the semantic continuity and semantic similarity between the two target sentences in the first sentence pair.
[0022] If the semantic association score is greater than the preset score, the last semantic segment obtained from the previous segmentation and the first semantic segment obtained from the current segmentation will be merged into one semantic segment; otherwise, the last semantic segment obtained from the previous segmentation and the first semantic segment obtained from the current segmentation will be treated as different semantic segments.
[0023] In an optional implementation, calculating the semantic similarity between the two target sentences in the first sentence pair includes: extracting feature vectors of the two target sentences respectively; calculating the cosine similarity between the feature vectors of the two target sentences, and using the calculated cosine similarity as the semantic similarity.
[0024] In an optional implementation, calculating the semantic similarity between the two target sentences in the first sentence pair includes: inputting the two target sentences into a multimodal multitask embedding model to obtain the semantic similarity between the two target sentences.
[0025] In an optional implementation, the semantic association score of the first sentence pair is calculated as follows: Semantic association score = Semantic continuity × Continuity weight + Semantic similarity × Similarity weight.
[0026] In an optional implementation, the step of semantically segmenting multiple target sentences within the preset sliding window to obtain at least one semantic segment includes: obtaining the semantic continuity of sentence pairs composed of two adjacent target sentences; calculating the semantic similarity of sentence pairs composed of two adjacent target sentences; calculating the semantic association score of sentence pairs composed of two adjacent target sentences based on the semantic continuity and semantic similarity between the two target sentences in the sentence pairs; if the semantic association score is greater than a preset score, then the target sentences in the corresponding sentence pairs are assigned to the same semantic segment; otherwise, they are assigned to different semantic segments.
[0027] In an optional implementation, after the step of clustering the semantic segments obtained from each segmentation with the semantic segments obtained from the previous segmentation to obtain the target semantic segments to which each target sentence within the preset sliding window belongs after the previous sliding and the current sliding, the method further includes:
[0028] Obtain preset constraints, which are configured to restrict the retrieval of semantic segments from the input of large models in the enhanced RAG system;
[0029] According to the preset constraints, each target semantic segment is processed to obtain a reference semantic segment corresponding to each target semantic segment, and each reference semantic segment is stored in a vector database in a vectorized form. When the large model receives input data, it retrieves the semantic segment vector related to the input data from the vector database and provides feedback based on the input data and the retrieved semantic segment vector.
[0030] In an optional implementation, the preset constraints include the maximum length of a single semantic segment of the large model input and / or the overlap between two adjacent semantic segments of the large model input.
[0031] In an optional implementation, the preset constraint is a preset length, where the sum of the lengths of all sentences in the target semantic segment is greater than the preset length; the step of processing each target semantic segment according to the preset constraint to obtain a reference semantic segment corresponding to each target semantic segment includes:
[0032] For any of the target semantic segments, if the target semantic segment does not meet the preset restriction conditions, the target semantic segment is further segmented according to the preset restriction conditions to obtain the reference semantic segment corresponding to the target semantic segment;
[0033] If the target semantic segment meets the preset restriction conditions, then the target semantic segment is used as its corresponding reference semantic segment.
[0034] In an optional implementation, the step of further segmenting the target semantic segment according to the preset constraints to obtain the reference semantic segment corresponding to the target semantic segment includes:
[0035] The first sentence in the target semantic segment is used as the reference sentence;
[0036] Starting from the reference sentence, a target sentence sequence is determined from the target semantic segment, wherein the sum of the lengths of all sentences in the target sentence sequence is the largest and less than the preset length;
[0037] All sentences in the target sentence sequence are grouped into a single reference semantic segment;
[0038] If the last sentence in the target sentence sequence is not the last sentence of the target semantic segment, then a reference sentence is re-determined from the target semantic segment, and the step of determining the target sentence sequence from the target semantic segment starting from the reference sentence is returned until all reference semantic segments corresponding to the target semantic segment are obtained.
[0039] In an optional implementation, the step of re-determining the reference sentence from the target semantic segment includes:
[0040] Determine the first related sentence from the target sentence sequence;
[0041] Determine the second related sentence from the sentences following the target sentence sequence in the target semantic segment;
[0042] If the sum of the lengths of the first associated sentence and the second associated sentence is greater than the preset length, then the reference sentence is determined based on the second associated sentence; otherwise, the reference sentence is determined based on the first associated sentence.
[0043] Secondly, this disclosure provides a text segmentation apparatus, the apparatus comprising:
[0044] The acquisition module is configured to acquire multiple sentences obtained after sentence-level segmentation of the target text.
[0045] The segmentation module is configured to take the first sentence of the plurality of sentences as the starting point and the last sentence as the ending point, slide a preset sliding window multiple times, and after each slide, segment the plurality of target sentences in the preset sliding window according to semantics to obtain at least one semantic segment;
[0046] The clustering module is configured to cluster the semantic segments obtained from each segmentation with the semantic segments obtained from the previous segmentation, thereby obtaining the target semantic segments to which each target sentence belongs within the preset sliding window after the previous slide and the current slide, until the semantic segments to which each sentence belongs among the multiple sentences are obtained.
[0047] Thirdly, this disclosure provides an electronic device including a processor and a memory, the memory being configured to store a program, and the processor being configured to implement the text segmentation method as described in any of the foregoing embodiments when executing the program.
[0048] Fourthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the text segmentation method as described in any of the foregoing embodiments.
[0049] Compared to existing technologies, the text segmentation method, apparatus, electronic device, and computer-readable storage medium disclosed in this disclosure provide a method for segmenting text into multiple sentences by sliding a preset sliding window and sequentially segmenting the target sentences within the preset sliding window according to semantics. This ensures the semantic coherence of the segmented target sentences within the preset sliding window. Furthermore, the semantic segments obtained from the current segmentation and those obtained from the previous segmentation are clustered to ensure the semantic coherence of sentences between adjacent sliding windows. Ultimately, this achieves reasonable text segmentation, ensures the semantic coherence of the segmented semantic segments, and thus improves the accuracy and reliability of the question-and-answer output of RAG feedback. Attached Figure Description
[0050] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this disclosure and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0051] Figure 1 is a framework diagram of the RAG system provided in the embodiments of this disclosure.
[0052] Figure 2 is a flowchart illustrating the text segmentation method provided in this embodiment of the present disclosure.
[0053] Figure 3 is an example diagram of the semantic segmentation process using a preset sliding window provided in an embodiment of this disclosure.
[0054] Figure 4 is a block diagram of a text segmentation device provided in an embodiment of this disclosure.
[0055] Figure 5 is a block diagram of an electronic device provided in an embodiment of this disclosure.
[0056] Icons: 10-Electronic device; 11-Processor; 12-Memory; 13-Bus; 100-Text segmentation device; 110-Acquisition module; 120-Segmentation module; 130-Clustering module. Detailed Implementation
[0057] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. The components of the embodiments of this disclosure described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.
[0058] Therefore, the following detailed description of the embodiments of this disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely to illustrate selected embodiments of the disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.
[0059] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.
[0060] In the description of this disclosure, it should be noted that if terms such as "upper," "lower," "inner," or "outer" are used to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the disclosed product is usually placed, they are only for the convenience of describing this disclosure and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this disclosure.
[0061] Furthermore, the use of terms such as "first" and "second" is merely for distinguishing descriptions and should not be interpreted as indicating or implying relative importance.
[0062] It should be noted that, where there is no conflict, the features in the embodiments of this disclosure can be combined with each other.
[0063] In existing RAG systems, relevant information is first retrieved from external documents before generating a response. This retrieved knowledge is then used as context input into the large model, thereby enhancing the model's knowledge coverage and improving its accuracy and reliability. Please refer to Figure 1, which is a framework diagram of the RAG system provided in this embodiment. The RAG system includes the following processing steps:
[0064] (1) Perform text segmentation on the document to obtain the segmented text segments;
[0065] (2) Vectorize and store the segmented text segments to generate a vector database;
[0066] (3) When processing a specified problem, perform vector retrieval on the vector database, retrieve vectors associated with the specified problem from the vector database, restore the retrieved vectors to the corresponding text segments (semantic segments in the figure), and use them as the context of the specified problem. Based on the specified problem and its context, construct the prompt words (Prompt) of the large model in the RAG system.
[0067] (4) Output questions and answers based on prompts.
[0068] In the RAG framework, document segmentation typically employs a fixed segmentation granularity. This granularity is usually associated with the input length limit of the large model. While this fixed granularity can meet the input requirements of large models, it inevitably leads to semantic fragmentation. Furthermore, the input length limit varies for different large models, making the fixed-granularity segmentation method inflexible and unable to adapt to the needs of different scenarios.
[0069] In view of this, the present disclosure provides a text segmentation method, apparatus, electronic device, and computer-readable storage medium, which can reasonably segment text according to semantics and ensure the semantic coherence of the segmented semantic segments. These will be described in detail below.
[0070] Please refer to Figure 2, which is a flowchart illustrating a text segmentation method provided in this embodiment of the present disclosure. The method includes the following steps:
[0071] Step S101: Obtain multiple sentences obtained after sentence-level segmentation of the target text.
[0072] In this embodiment of the disclosure, the target text may be text extracted from one or more documents, such as Word documents, TXT documents, PDF documents, etc., or text extracted from images after text recognition.
[0073] In this embodiment of the disclosure, a pre-trained model can be used to segment the target text at the sentence level, or the target text can be segmented at the sentence level based on punctuation marks and line breaks in the target text.
[0074] Step S102: Starting from the first sentence of a plurality of sentences and ending with the last sentence, slide a preset sliding window multiple times. After each slide, the plurality of target sentences within the preset sliding window are semantically segmented to obtain at least one semantic segment.
[0075] In this embodiment of the disclosure, the size of the preset sliding window affects the segmentation efficiency and segmentation accuracy. The window size represents the number of sentences that need to be semantically segmented each time. The size of the preset sliding window can be adjusted according to the business application scenario. For example, the preset sliding window size is 2.
[0076] In this embodiment of the disclosure, semantic segmentation can be performed using a pre-trained language model. The language model includes, but is not limited to, the BERT-NSP model (a deep learning model that uses the Transformer architecture to process natural language processing), the BGE model (a text embedding model configured to convert text into low-dimensional dense vectors for efficient semantic analysis and retrieval), and the SimCSE (Similarity Consensus Semantic Encoding) model (a model configured to generate sentence embeddings), etc.
[0077] Step S103: For each semantic segment obtained from the segmentation, the semantic segments obtained from the current segmentation and the semantic segments obtained from the previous segmentation are clustered to obtain the target semantic segments to which each target sentence belongs within the preset sliding window after the previous sliding and after the current sliding, until the semantic segments to which each sentence belongs in multiple sentences are obtained.
[0078] In this embodiment of the disclosure, in order to ensure semantic coherence between the semantic segments obtained from the two segments, if the last semantic segment of the previous segment and the first semantic segment of the current segment are semantically coherent, the two semantic segments need to be merged into one semantic segment; otherwise, they are treated as two independent semantic segments. For example, if the last semantic segment 1 of the previous segment includes sentence a and sentence b, and the first semantic segment 2 of the current segment includes sentence b, sentence c, and sentence d, then semantic segment 1 and semantic segment 2 are semantically coherent, and therefore semantic segment 1 and semantic segment 2 can be merged into one semantic segment.
[0079] In this embodiment of the disclosure, as an optional implementation, clustering can be performed once by sliding once, or multiple times can be slid and then the semantic segments obtained by semantic segmentation after each slide can be clustered successively, and finally the semantic segments to which each sentence belongs in multiple sentences can be obtained.
[0080] The method provided in this embodiment sequentially segments the target sentence within a preset sliding window according to semantics, ensuring the semantic coherence of the segmented target sentence within the preset sliding window. Furthermore, it clusters the semantic segments obtained from the current segmentation and the semantic segments obtained from the previous segmentation to ensure the semantic coherence of sentences between the preset sliding windows after two adjacent slides, thus achieving reasonable text segmentation.
[0081] In an optional implementation, to segment multiple target sentences semantically, segmentation can be performed by determining whether adjacent target sentences are semantically continuous. One optional implementation is as follows:
[0082] First, multiple target sentences are input into a pre-trained text segmentation model to obtain the semantic continuity of sentence pairs composed of two adjacent target sentences;
[0083] In this embodiment, the text segmentation model can be based on BERT-NSP, BGE, and SimCSE (Similarity Consensus Semantic Encoding). Taking the BERT-NSP-based text segmentation model as an example, during the construction of the training dataset, misalignment annotations can be performed on sentences based on the document content. That is, when processing the document, sentences that are inconsistent with the overall document content or contain errors are identified and annotated. This allows for automatic error correction in the document based on misalignment annotations, improving the accuracy of information extraction from the document, and accurately evaluating the overall quality and consistency of the document.
[0084] In addition, when samples are scarce between paragraphs in the dataset, an oversampling strategy can be used, which involves increasing the number of samples in the minority class to balance the number of samples in each class. Common oversampling strategies include, but are not limited to, random oversampling and interpolating between samples in the minority class to generate new synthetic samples, in order to ensure the sample balance of the training data and thus improve the generalization ability of the model.
[0085] Secondly, for any pair of target sentences, if the semantic continuity of the pair of target sentences is greater than the preset value, then the two target sentences in the pair of target sentences are assigned to the same semantic segment; otherwise, the two target sentences in the pair of target sentences are assigned to different semantic segments.
[0086] In this embodiment of the disclosure, as an optional implementation, the semantic continuity obtained by the text segmentation model can be a probability value of semantic continuity between two target sentences. If it is greater than a preset value, the two target sentences are considered to belong to the same semantic segment; otherwise, they are considered to belong to different semantic segments. As another optional implementation, the semantic continuity obtained by the text segmentation model can be a string composed of 0 and / or 1, where 0 indicates semantic discontinuity between the corresponding two target sentences, and 1 indicates semantic continuity between the corresponding two target sentences. For example, multiple target sentences are sentence a, sentence b, sentence c, and sentence d. Sentence a and sentence b form sentence pair 1, sentence b and sentence c form sentence pair 2, and sentence c and sentence d form sentence pair 3. The string output by the text segmentation model is 011. Here, "0" represents the semantic continuity of sentence pair 1, indicating semantic discontinuity between sentence a and sentence b; the first "1" represents the semantic continuity of sentence pair 2, indicating semantic continuity between sentence b and sentence c; and the second "1" represents the semantic continuity of sentence pair 3, indicating semantic continuity between sentence c and sentence d.
[0087] Finally, each sentence pair is used as a target sentence pair in turn to obtain at least one semantic segment after semantic segmentation of multiple target sentences.
[0088] In this embodiment of the disclosure, semantically continuous sentences in multiple target sentences are divided into the same semantic segment, and semantically discontinuous sentences are divided into different semantic segments, thereby obtaining at least one semantic segment after semantic segmentation of multiple target sentences.
[0089] In an optional implementation, to ensure semantic coherence between adjacent segments, the last target sentence in the previous preset sliding window is used as the first target sentence in the current preset sliding window. That is, the first target sentence in the current preset sliding window is the same as the last target sentence in the previous preset sliding window. Based on this, one method for clustering the semantic segments obtained in the current segmentation and the semantic segments obtained in the previous segmentation is as follows:
[0090] First, obtain the semantic continuity of the first sentence pair in the preset sliding window after this sliding;
[0091] In this embodiment of the disclosure, the semantic continuity of the first sentence pair can be obtained by inputting the two sentences in the first sentence pair into the text segmentation model described above.
[0092] Secondly, the semantic similarity between the two target sentences in the first sentence pair is calculated;
[0093] In this embodiment of the disclosure, one optional implementation is to obtain semantic similarity by inputting two target sentences into models such as BGE or M3E (Multi-Modal Multi-Task Embedding). Another implementation is to extract feature vectors of the two target sentences respectively, calculate the cosine similarity between the two feature vectors, and use the calculated cosine similarity as the semantic similarity.
[0094] Third, the semantic association score of the first sentence pair is calculated based on the semantic continuity and semantic similarity between the two target sentences in the first sentence pair;
[0095] In this embodiment of the disclosure, a continuity weight and a similarity weight can be set for semantic continuity and semantic similarity, respectively. The semantic association score = semantic continuity × continuity weight + semantic similarity × similarity weight. The semantic continuity weight and semantic similarity weight can be set according to the actual application scenario and the reliability of their respective models or the degree of matching with the actual application scenario. For example, the continuity weight can be set to 0.7 and the similarity weight can be set to 0.3.
[0096] Finally, if the semantic association score is greater than the preset score, the last semantic segment obtained from the previous segmentation and the first semantic segment obtained from the current segmentation will be merged into one semantic segment; otherwise, the last semantic segment obtained from the previous segmentation and the first semantic segment obtained from the current segmentation will be treated as different semantic segments.
[0097] In this embodiment of the disclosure, the preset score can be set according to actual needs, for example, the preset score is 0.5.
[0098] It should be noted that, where semantic segmentation efficiency allows, semantic association scores can also be used to semantically segment multiple target sentences within a preset sliding window after each slide, obtaining at least one semantic segment. Another way to achieve this is as follows:
[0099] Obtain the semantic continuity of sentence pairs consisting of two adjacent target sentences;
[0100] Calculate the semantic similarity of sentence pairs consisting of two adjacent target sentences;
[0101] The semantic association score of the sentence pair composed of two adjacent target sentences is calculated based on the semantic continuity and semantic similarity between the two target sentences in the sentence pair.
[0102] If the semantic association score is greater than the preset score, the target sentences in the corresponding sentence pair will be classified into the same semantic segment; otherwise, they will be classified into different semantic segments.
[0103] Furthermore, to improve the efficiency of semantic segmentation, the semantic similarity of the unconfirmed semantic pairs in multiple target sentences with a semantic continuity less than or equal to a preset value can be calculated. Based on the semantic continuity and semantic similarity of the unconfirmed semantic pairs, a semantic association score is calculated. If the semantic association score is greater than a preset value, the target sentences in the unconfirmed semantic pair are assigned to the same semantic segment; otherwise, they are assigned to different semantic segments. Since this method only targets unconfirmed semantic pairs with a semantic continuity less than or equal to a preset value and determines whether the target sentences included in them belong to the same semantic segment by calculating a semantic association score, the computational load is reduced to some extent. Compared with the method of calculating the semantic association score of each sentence pair in the target sentence, this method improves the efficiency of semantic segmentation to some extent.
[0104] To more intuitively demonstrate the process of semantic segmentation of multiple sentences during the sliding of a preset sliding window, please refer to Figure 3. Figure 3 is an example diagram of the process of semantic segmentation using a preset sliding window provided in this embodiment of the present disclosure. For sentences 1, 2, and 3 within the preset sliding window, semantic segmentation is performed to obtain: semantic segment [sentence 1, sentence 2] and semantic segment [sentence 3]. Sliding the preset sliding window, semantic segmentation is performed for sentences 3, 4, and 5 within the preset sliding window to obtain: semantic segment [sentence 3, sentence 4, sentence 5]. The semantic segment [sentence 3, sentence 4, sentence 5] obtained this time is clustered with the semantic segment [sentence 1, sentence 2] and semantic segment [sentence 3] obtained last time to obtain semantic segment [sentence 1, sentence 2] and semantic segment [sentence 3, sentence 4, sentence 5].
[0105] To demonstrate the effectiveness of the semantic segmentation methods provided in this embodiment, this embodiment segments 19 manually annotated sentences using three segmentation methods and evaluates and compares the segmentation results of the three methods. Method 1: Fixed character count segmentation; Method 2: Segmentation using a preset sliding window and the BERT-NSP model in this embodiment; Method 3: Segmentation using a preset sliding window and the BERT-NSP+M3E model in this embodiment.
[0106] Segmentation results of Method 1: 18 segments, with 2 correctly segmented segments; P = 11.11%, R = 94.73%, F1 = 19.87%;
[0107] The segmentation results of Method 2: 16 segments, with 13 correctly segmented segments, P = 81.25%, R = 84.21%, F1 = 82.70%;
[0108] The segmentation results of Method 3: 18 segments, with 16 segments correctly segmented, P = 88.88%, R = 94.73%, F1 = 91.71%;
[0109] Where P represents precision, calculated as: number of correctly segmented segments / total number of segments; R represents recall, calculated as: number of correctly segmented segments / total number of segments that should be segmented; F1 represents the segmentation score, calculated as: 2 × precision × recall / (precision + recall).
[0110] In this embodiment of the disclosure, in order to enable the segmented semantic segments to flexibly adapt to the input requirements of different large models, after obtaining the target semantic segments to which each target sentence belongs within the preset sliding window after the last slide and the current slide belongs, it is also necessary to process the target semantic segments according to the input requirements of the required large model. This embodiment of the disclosure provides a processing method:
[0111] First, obtain the preset constraints, which are configured to restrict the semantic segments of the large model input in the RAG system for retrieval enhancement.
[0112] In this embodiment of the disclosure, the preset limiting condition can be the maximum length of a single semantic segment of the large model input, for example, the maximum length of each semantic segment does not exceed 500 characters, or it can be the overlap between two adjacent semantic segments of the large model input, for example, the first semantic segment is from sentence 1 to sentence 10, and the second semantic segment is from sentence 6 to sentence 15.
[0113] It should be noted that the preset constraints can also combine the maximum length and the degree of overlap, that is, to limit the length of a single semantic segment in the input of a large model, while also limiting the degree of overlap between two adjacent semantic segments.
[0114] Secondly, based on preset constraints, each target semantic segment is processed to obtain a reference semantic segment corresponding to each target semantic segment. Each reference semantic segment is then stored in a vector database in a vectorized form. When the large model receives input data, it retrieves the semantic segment vectors related to the input data from the vector database and provides feedback based on the input data and the retrieved semantic segment vectors.
[0115] In this embodiment of the disclosure, a reference semantic segment is a semantic segment that meets preset constraints. One target semantic segment can correspond to one reference semantic segment, or one target semantic segment can correspond to multiple reference semantic segments. When one target semantic segment corresponds to multiple reference semantic segments, in order to ensure semantic coherence among the multiple reference semantic segments, there may be partial overlap between the multiple reference semantic segments.
[0116] In an optional implementation, for large models with limited input length, in order to meet the length requirements of the large model while maintaining semantic coherence between sentences as much as possible, this disclosure provides a method for processing the target semantic segment to obtain a reference semantic segment that meets the input length requirements of the large model:
[0117] First, for any target semantic segment, if the target semantic segment does not meet the preset constraints, the target semantic segment is further segmented according to the preset constraints to obtain the reference semantic segment corresponding to the target semantic segment;
[0118] Secondly, if the target semantic segment meets the preset constraints, then the target semantic segment is used as its corresponding reference semantic segment.
[0119] In this embodiment, the preset constraint is a preset length. If the target semantic segment meets the preset constraint, it means that the sum of the lengths of all sentences in the target semantic segment is less than or equal to the preset length. In this case, no processing is required on the target semantic segment, and the target semantic segment becomes the reference semantic segment. Otherwise, it means that the sum of the lengths of all sentences in the target semantic segment is greater than the preset length. In this case, the target semantic segment needs to be further segmented so that the segmented semantic segments meet the preset constraint. One way to further segment is as follows:
[0120] (1) Use the first sentence in the target semantic segment as the reference sentence;
[0121] (2) Starting from the reference sentence, determine the target sentence sequence from the target semantic segment, wherein the sum of the lengths of all sentences in the target sentence sequence is the largest and less than the preset length;
[0122] (3) Group all sentences in the target sentence sequence into a single reference semantic segment;
[0123] (4) If the last sentence in the target sentence sequence is not the last sentence of the target semantic segment, then the reference sentence is re-determined from the target semantic segment and the process is returned to step (2) until all reference semantic segments corresponding to the target semantic segment are obtained.
[0124] In an optional implementation, in order to ensure the semantic coherence between the final reference semantic segments as much as possible during re-segmentation, this disclosure provides an implementation method for determining reference sentences:
[0125] First, identify the first relevant sentence from the target sentence sequence;
[0126] Secondly, the second related sentence is determined from the sentence following the target sentence sequence in the target semantic segment;
[0127] Finally, if the sum of the lengths of the first and second related sentences is greater than the preset length, the reference sentence is determined based on the second related sentence; otherwise, the reference sentence is determined based on the first related sentence.
[0128] In this embodiment of the disclosure, the reference sentence is determined based on the second associated sentence in either the first sentence of the second associated sentence or the first sentence of the last few sentences of the first associated sentence. As long as the sum of the lengths of the last few sentences of the first associated sentence and the second associated sentence is less than or equal to the preset length, it is acceptable.
[0129] In this embodiment of the disclosure, the number of the first associated sentence and the second associated sentence can both be one or more, depending on the maximum allowed overlap during segmentation. The first associated sentence can be a sentence located at the end of a preset number of sentences in the target sentence sequence. As one implementation, the preset number can be fixed, depending on the maximum allowed overlap during segmentation; for example, the preset number can be set to 2. As another implementation, the preset number can also be variable, as long as the sum of the lengths of the first associated sentence and the second associated sentence is less than or equal to the preset length.
[0130] This embodiment of the disclosure uses an example where the number of first associated sentences is 2 and the number of second associated sentences is 1. For example, the target semantic segment includes the following sentences:
[0131] Sentence 1, Sentence 2, Sentence 3, Sentence 4, Sentence 5, Sentence 6, Sentence 7
[0132] Taking sentence 1 as the reference sentence, the sum of the lengths of sentences 1, 2, and 3 is the largest and less than the preset length. Then the target sentence sequence is: (sentence 1, sentence 2, sentence 3), and sentences 1, 2, and 3 belong to reference semantic segment 1.
[0133] Sentences 2 and 3 are designated as the first related sentences, and sentence 4 as the second related sentence. If the sum of the lengths of sentences 2, 3, and 4 is less than the preset length, then sentence 2 is designated as the reference sentence.
[0134] If the sum of the lengths of sentences 2, 3, 4, and 5 is the largest and less than the preset length, then the target sentence sequence is: (sentence 2, sentence 3, sentence 4, sentence 5), and sentences 2, 3, 4, and 5 belong to reference semantic segment 2;
[0135] Sentences 4 and 5 are designated as the first related sentences, and sentence 6 as the second related sentence. If the sum of the lengths of sentences 4, 5, and 6 is greater than the preset length, then sentence 6 can be used as the reference sentence. If the sum of the lengths of sentences 5 and 6 is less than the preset length, then sentence 5 can also be used as the reference sentence. In this example, sentence 5 is used as the reference sentence.
[0136] If the sum of the lengths of sentences 5, 6, and 7 is less than the preset length, then the target sentence sequence is (sentence 5, sentence 6, sentence 7), and sentences 5, 6, and 7 belong to reference semantic segment 3.
[0137] Therefore, the target semantic segment is further divided into 3 reference semantic segments: reference semantic segment 1 to reference semantic segment 3.
[0138] In this embodiment of the disclosure, the preset limiting condition can be either a preset length or an overlap number. With an overlap number of 2, the target semantic segment [sentence 1, sentence 2, sentence 3, sentence 4, sentence 5] can be further divided into two reference semantic segments: (sentence 1, sentence 2, sentence 3, sentence 4) and (sentence 3, sentence 4, sentence 5). The two reference semantic segments together contain two overlapping sentences: sentence 3 and sentence 4. Alternatively, they can be further divided into (sentence 1, sentence 2, sentence 3), (sentence 2, sentence 3, sentence 4), and (sentence 3, sentence 4, sentence 5), depending on actual needs.
[0139] To perform the corresponding steps in the above embodiments and various possible implementations, an implementation method of the text segmentation device 100 is given below. Please refer to Figure 4, which is a block diagram of the text segmentation device provided in the embodiments of this disclosure. It should be noted that the basic principle and technical effects of the text segmentation device 100 provided in this disclosure are the same as those in the corresponding embodiments described above. For the sake of brevity, the embodiments of this disclosure do not mention it.
[0140] The text segmentation device 100 includes an acquisition module 110, a segmentation module 120, and a clustering module 130.
[0141] The acquisition module 110 is configured to acquire multiple sentences obtained after sentence-level segmentation of the target text.
[0142] The segmentation module 120 is configured to take the first sentence of multiple sentences as the starting point and the last sentence as the ending point, slide a preset sliding window multiple times, and after each slide, segment the multiple target sentences in the preset sliding window according to semantics to obtain at least one semantic segment.
[0143] The clustering module 130 is configured to cluster the semantic segments obtained in each segmentation and the semantic segments obtained in the previous segmentation to obtain the target semantic segments to which each target sentence belongs in the preset sliding window after the previous sliding and the current sliding, until the semantic segments to which each sentence belongs in multiple sentences are obtained.
[0144] In an optional implementation, the segmentation module 120 is configured to:
[0145] Multiple target sentences are input into a pre-trained text segmentation model to obtain the semantic continuity of sentence pairs composed of two adjacent target sentences;
[0146] For any pair of target sentences, if the semantic continuity of the pair of target sentences is greater than a preset value, then the two target sentences in the pair of target sentences are assigned to the same semantic segment; otherwise, the two target sentences in the pair of target sentences are assigned to different semantic segments.
[0147] Each sentence pair is used as a target sentence pair in turn to obtain at least one semantic segment after semantic segmentation of multiple target sentences.
[0148] In an optional implementation, the first target sentence in the preset sliding window after this slide is the same as the last target sentence in the preset sliding window after the previous slide, and the two adjacent target sentences in the preset sliding window form a sentence pair;
[0149] For each semantic segment obtained from the segmentation, clustering module 130 is configured as follows:
[0150] Obtain the semantic continuity of the first sentence pair within the preset sliding window after this sliding action;
[0151] Calculate the semantic similarity between the two target sentences in the first sentence pair;
[0152] The semantic association score of the first sentence pair is calculated based on the semantic continuity and semantic similarity between the two target sentences in the first sentence pair.
[0153] If the semantic association score is greater than the preset score, the last semantic segment obtained from the previous segmentation and the first semantic segment obtained from the current segmentation will be merged into one semantic segment; otherwise, the last semantic segment obtained from the previous segmentation and the first semantic segment obtained from the current segmentation will be treated as different semantic segments.
[0154] In an optional implementation, the clustering module 130 is further configured to:
[0155] Obtain preset constraints, and configure the preset constraints to limit the retrieval of semantic segments that are input to large models in the enhanced RAG system;
[0156] Based on preset constraints, each target semantic segment is processed to obtain a reference semantic segment corresponding to each target semantic segment. Each reference semantic segment is then stored in a vector database in a vectorized form. When the large model receives input data, it retrieves the semantic segment vectors related to the input data from the vector database and provides feedback based on the input data and the retrieved semantic segment vectors.
[0157] In an optional implementation, the preset constraint is a preset length, where the sum of the lengths of all sentences in the target semantic segment is greater than the preset length; the clustering module 130 is configured to process each target semantic segment according to the preset constraint to obtain a reference semantic segment corresponding to each target semantic segment as follows:
[0158] For any target semantic segment, if the target semantic segment does not meet the preset constraints, the target semantic segment is further segmented according to the preset constraints to obtain the reference semantic segment corresponding to the target semantic segment;
[0159] If the target semantic segment meets the preset constraints, then the target semantic segment will be used as its corresponding reference semantic segment.
[0160] In an optional implementation, when the clustering module 130 is configured to further segment the target semantic segment according to preset constraints to obtain the reference semantic segment corresponding to the target semantic segment, it is configured as follows:
[0161] Use the first sentence in the target semantic segment as the reference sentence;
[0162] Starting from the reference sentence, determine the target sentence sequence from the target semantic segment, where the sum of the lengths of all sentences in the target sentence sequence is the largest and less than the preset length;
[0163] All sentences in the target sentence sequence are grouped into a single reference semantic segment;
[0164] If the last sentence in the target sentence sequence is not the last sentence of the target semantic segment, then a new reference sentence is determined from the target semantic segment, and the process of determining the target sentence sequence from the target semantic segment starting from the reference sentence is repeated until all reference semantic segments corresponding to the target semantic segment are obtained.
[0165] In an optional implementation, clustering module 130 is configured, when configured to redetermine reference sentences from the target semantic segment, to:
[0166] Identify the first related sentence from the target sentence sequence;
[0167] Identify the second related sentence from the sentences following the target sentence sequence in the target semantic segment;
[0168] If the sum of the lengths of the first and second related sentences is greater than the preset length, then the reference sentence is determined based on the second related sentence; otherwise, the reference sentence is determined based on the first related sentence.
[0169] This disclosure also provides a block diagram of an electronic device 10. The electronic device 10 implements the text segmentation method of the aforementioned embodiments. Please refer to Figure 5. Figure 5 is a block diagram of the electronic device 10 provided in this disclosure. The electronic device 10 includes a processor 11, a memory 12 and a bus 13. The processor 11 and the memory 12 are connected through the bus 13.
[0170] Processor 11 can be an integrated circuit chip with signal processing capabilities. In implementation, each step of the text segmentation method described in the above embodiments can be completed by the integrated logic circuits in the hardware of processor 11 or by software instructions. Processor 11 can be a general-purpose processor, including CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Logic Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0171] The memory 12 is configured to store a program that implements the text segmentation method. This program may be a software function module stored in the memory 12 in the form of software or firmware or embedded in the OS (Operating System) of the electronic device 10.
[0172] After receiving the execution instruction, the processor 11 executes the program to implement the text segmentation method of the aforementioned embodiment.
[0173] This disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the text segmentation method as described above.
[0174] In summary, the embodiments of this disclosure provide a text segmentation method, apparatus, electronic device, and computer-readable storage medium. The method includes: acquiring multiple sentences obtained by performing sentence-level segmentation on target text; sliding a preset sliding window multiple times, starting from the first sentence and ending at the last sentence, and after each slide, semantically segmenting the multiple target sentences within the preset sliding window to obtain at least one semantic segment; clustering the semantic segment obtained in each segmentation with the semantic segment obtained in the previous segmentation to obtain the target semantic segment to which each target sentence in the preset sliding window belongs after the previous slide and the current slide, until the semantic segment to which each sentence in the multiple sentences belongs is obtained. Compared with the prior art, the present disclosure has at least the following advantages: (1) For multiple sentences, a preset sliding window is slid, and the target sentences in the preset sliding window are sequentially segmented according to semantics to ensure the semantic coherence of the target sentences after segmentation in the preset sliding window. Furthermore, the semantic segments obtained from this segmentation and the semantic segments obtained from the previous segmentation are clustered to ensure the semantic coherence of sentences between the preset sliding windows after two adjacent slides. Finally, reasonable text segmentation is achieved, ensuring the semantic coherence of the segmented semantic segments, thereby improving the accuracy and reliability of the question-and-answer output of RAG feedback; (2) The semantic continuity and semantic coherence between sentences in the preset sliding window after two adjacent slides are utilized. (2) Semantic similarity, accurately judge the semantic coherence between the two, and improve the rationality of semantic segmentation; (3) Since the semantic coherence between sentences in the semantic segment after semantic segmentation is guaranteed, the hardware resource requirements of the large model of RAG are reduced, and it is more universal and economical, so as to achieve the purpose of cost reduction and efficiency improvement; (4) While segmenting, it can also take into account the preset restriction conditions of the semantic segment of the input of the large model in the RAG system. The semantic segment obtained by segmentation is processed according to the preset restriction conditions, so that the final segmentation result can meet the needs of the large model in the RAG system while ensuring semantic coherence to the greatest extent, so as to achieve the effect of flexibly adapting to the requirements of different large models.
[0175] The above descriptions are merely various embodiments of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims. Industrial applicability
[0176] The text segmentation method, apparatus, electronic device, and computer-readable storage medium provided in this disclosure include: acquiring multiple sentences obtained by performing sentence-level segmentation on target text; sliding a preset sliding window multiple times, starting from the first sentence and ending at the last sentence among the multiple sentences, and after each slide, semantically segmenting the multiple target sentences within the preset sliding window to obtain at least one semantic segment; clustering the semantic segment obtained in this segmentation with the semantic segment obtained in the previous segmentation for each semantic segment obtained in the previous segmentation to obtain the target semantic segment to which each target sentence in the preset sliding window belongs after the previous slide and the current slide, until the semantic segment to which each sentence in the multiple sentences belongs is obtained. The present invention has the following advantages: (1) For multiple sentences, a preset sliding window is slid, and the target sentences in the preset sliding window are sequentially segmented according to semantics to ensure the semantic coherence of the target sentences after segmentation in the preset sliding window. Furthermore, the semantic segments obtained in this segmentation and the semantic segments obtained in the previous segmentation are clustered to ensure the semantic coherence of sentences between the preset sliding windows after two adjacent slides. Finally, reasonable segmentation of text is achieved, ensuring the semantic coherence of the segmented semantic segments, thereby improving the accuracy and reliability of the question-and-answer output of RAG feedback; (2) The semantic continuity and semantic similarity between sentences in the preset sliding window after two adjacent slides are utilized. (3) Accurately judge the semantic coherence between the two and improve the rationality of semantic segmentation; (4) Since the semantic coherence between sentences in the semantic segmentation is guaranteed, the hardware resource requirements of the large model of RAG are reduced, and it is more universal and economical, so as to achieve the purpose of cost reduction and efficiency improvement; (5) While segmenting, it can also take into account the preset restriction conditions of the semantic segment of the input of the large model in the RAG system. The semantic segment obtained by segmentation is processed according to the preset restriction conditions, so that the final segmentation result can meet the needs of the large model in the RAG system while ensuring semantic coherence to the greatest extent, so as to achieve the effect of flexibly adapting to the requirements of different large models.
Claims
1. A text segmentation method, characterized in that, The method includes: Obtain multiple sentences obtained after sentence-level segmentation of the target text; Starting with the first sentence and ending with the last sentence among the multiple sentences, a preset sliding window is slid multiple times. After each slide, the multiple target sentences within the preset sliding window are semantically segmented to obtain at least one semantic segment. For each semantic segment obtained from the segmentation, the semantic segments obtained from the current segmentation and the semantic segments obtained from the previous segmentation are clustered to obtain the target semantic segments to which each target sentence belongs within the preset sliding window after the previous sliding and after the current sliding, until the semantic segments to which each sentence belongs among the multiple sentences are obtained.
2. The text segmentation method according to claim 1, characterized in that, The step of semantically segmenting multiple target sentences within the preset sliding window to obtain at least one semantic segment includes: The multiple target sentences are input into a pre-trained text segmentation model to obtain the semantic continuity of sentence pairs composed of two adjacent target sentences; For any pair of target sentences, if the semantic continuity of the pair of target sentences is greater than a preset value, then the two target sentences in the pair of target sentences are assigned to the same semantic segment; otherwise, the two target sentences in the pair of target sentences are assigned to different semantic segments. Each sentence pair is sequentially used as the target sentence pair to obtain at least one semantic segment after semantic segmentation of the multiple target sentences.
3. The text segmentation method according to claim 1, characterized in that, After this slide, the first target sentence in the preset sliding window is the same as the last target sentence in the preset sliding window after the previous slide. The two adjacent target sentences in the preset sliding window form a sentence pair. The step of clustering the semantic segments obtained from each segmentation with the semantic segments obtained from the previous segmentation to obtain the target semantic segment to which each target sentence in the preset sliding window belongs after the previous sliding and the current sliding includes: Obtain the semantic continuity of the first sentence pair within the preset sliding window after this sliding action; Calculate the semantic similarity between the two target sentences in the first sentence pair; The semantic association score of the first sentence pair is calculated based on the semantic continuity and semantic similarity between the two target sentences in the first sentence pair. If the semantic association score is greater than the preset score, the last semantic segment obtained from the previous segmentation and the first semantic segment obtained from the current segmentation will be merged into one semantic segment; otherwise, the last semantic segment obtained from the previous segmentation and the first semantic segment obtained from the current segmentation will be treated as different semantic segments.
4. The text segmentation method according to claim 3, characterized in that, The calculation of the semantic similarity between the two target sentences in the first sentence pair includes: Extract the feature vectors of the two target sentences respectively; Calculate the cosine similarity of the feature vectors of two target sentences, and use the calculated cosine similarity as the semantic similarity.
5. The text segmentation method according to claim 3, characterized in that, The calculation of the semantic similarity between the two target sentences in the first sentence pair includes: Two target sentences are input into a multimodal multitask embedding model to obtain the semantic similarity between the two target sentences.
6. The text segmentation method according to claim 3, characterized in that, The formula for calculating the semantic association score of the first sentence pair is: Semantic association score = semantic continuity × continuity weight + semantic similarity × similarity weight.
7. The text segmentation method according to claim 1, characterized in that, The step of semantically segmenting multiple target sentences within the preset sliding window to obtain at least one semantic segment includes: Obtain the semantic continuity of sentence pairs consisting of two adjacent target sentences; Calculate the semantic similarity of sentence pairs consisting of two adjacent target sentences; The semantic association score of the sentence pair composed of two adjacent target sentences is calculated based on the semantic continuity and semantic similarity between the two target sentences in the sentence pair. If the semantic association score is greater than the preset score, the target sentences in the corresponding sentence pair will be classified into the same semantic segment; otherwise, they will be classified into different semantic segments.
8. The text segmentation method according to any one of claims 1-7, characterized in that, After the step of clustering the semantic segments obtained from each segmentation with the semantic segments obtained from the previous segmentation to obtain the target semantic segments to which each target sentence within the preset sliding window belongs after the previous sliding and the current sliding, the method further includes: Obtain preset constraints, which are configured to restrict the retrieval of semantic segments from the input of large models in the enhanced RAG system; According to the preset constraints, each target semantic segment is processed to obtain a reference semantic segment corresponding to each target semantic segment, and each reference semantic segment is stored in a vector database in a vectorized form. When the large model receives input data, it retrieves the semantic segment vector related to the input data from the vector database and provides feedback based on the input data and the retrieved semantic segment vector.
9. The text segmentation method according to claim 8, characterized in that, The preset constraints include the maximum length of a single semantic segment of the large model input and / or the overlap between two adjacent semantic segments of the large model input.
10. The text segmentation method according to claim 8, characterized in that, The preset constraint is a preset length, whereby the sum of the lengths of all sentences in the target semantic segment is greater than the preset length. The step of processing each target semantic segment according to the preset constraints to obtain a reference semantic segment corresponding to each target semantic segment includes: For any of the target semantic segments, if the target semantic segment does not meet the preset restriction conditions, the target semantic segment is further segmented according to the preset restriction conditions to obtain the reference semantic segment corresponding to the target semantic segment; If the target semantic segment meets the preset restriction conditions, then the target semantic segment is used as its corresponding reference semantic segment.
11. The text segmentation method according to claim 10, characterized in that, The step of further segmenting the target semantic segment according to the preset constraints to obtain the reference semantic segment corresponding to the target semantic segment includes: The first sentence in the target semantic segment is used as the reference sentence; Starting from the reference sentence, a target sentence sequence is determined from the target semantic segment, wherein the sum of the lengths of all sentences in the target sentence sequence is the largest and less than the preset length; All sentences in the target sentence sequence are grouped into a single reference semantic segment; If the last sentence in the target sentence sequence is not the last sentence of the target semantic segment, then a reference sentence is re-determined from the target semantic segment, and the step of determining the target sentence sequence from the target semantic segment starting from the reference sentence is returned until all reference semantic segments corresponding to the target semantic segment are obtained.
12. The text segmentation method according to claim 11, characterized in that, The step of re-determining the reference sentence from the target semantic segment includes: Determine the first related sentence from the target sentence sequence; Determine the second related sentence from the sentences following the target sentence sequence in the target semantic segment; If the sum of the lengths of the first associated sentence and the second associated sentence is greater than the preset length, then the reference sentence is determined based on the second associated sentence; otherwise, the reference sentence is determined based on the first associated sentence.
13. A text segmentation device, characterized in that, The device includes: The acquisition module is configured to acquire multiple sentences obtained after sentence-level segmentation of the target text. The segmentation module is configured to take the first sentence of the plurality of sentences as the starting point and the last sentence as the ending point, slide a preset sliding window multiple times, and after each slide, segment the plurality of target sentences in the preset sliding window according to semantics to obtain at least one semantic segment; The clustering module is configured to cluster the semantic segments obtained from each segmentation with the semantic segments obtained from the previous segmentation, thereby obtaining the target semantic segments to which each target sentence belongs within the preset sliding window after the previous slide and the current slide, until the semantic segments to which each sentence belongs among the multiple sentences are obtained.
14. An electronic device, characterized in that, The device includes a processor and a memory, the memory being configured to store a program, and the processor being configured to, when executing the program, implement the text segmentation method as described in any one of claims 1-12.
15. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the text segmentation method as described in any one of claims 1-12.