303 results about "Text segmentation" patented technology

The invention discloses a grassroots power supply enterprise compliance risk intelligent system and method based on big data analysis. The data acquisition unit is used for acquiring business operation data, historical violation records and policy and regulation update data of basic power supply enterprises to form a unified compliance data set. And the natural language processing unit performs text word segmentation and correlation analysis on the policy and regulation and violation record data, extracts key risk factors and labels compliance risk labels. And the risk feature construction unit performs multi-dimensional feature fusion on the business operation data and the compliance risk label data to generate a feature matrix for risk identification. And the intelligent risk assessment unit performs real-time analysis on the feature matrix by using a pre-trained machine learning model, identifies compliance risk categories and levels, and generates early warning information. According to the method, the accuracy of compliance risk identification is improved by using big data analysis and an intelligent algorithm, the compliance management cost of basic-level power supply enterprises is reduced, and the operation safety and compliance of the enterprises are improved.

The present application relates to a retrieval model training method and apparatus and a computer device. The method comprises: acquiring a knowledge document, and performing text segmentation processing on the knowledge document to obtain a text block set (501); on the basis of the text block set and summary information comprised in each text block, writing a question for each text block to obtain one or more question texts corresponding to each text block, separately combining the one or more question texts corresponding to each text block with the corresponding text block to obtain one or more data pairs corresponding to each text block and, on the basis of the one or more data pairs corresponding to each text block, generating a data pair set (502); and using the data pair set to train a first retrieval model so as to obtain a second retrieval model (503).

Systems and methods for semantic temporal segmentation based on topic recognition are disclosed. Text classification and segmentation may be used to index media content for subsequent searches. The method may include using a text classification model to semantically analyze sentences in a text file (such as a transcript) to determine topics with which sentences are associated. The output of the text classification model may be provided to a text segmentation model to enable more-accurate identification of text segments (e.g., paragraphs) within the text file. In some examples, the output of a text segmentation model is provided to a text classification model to enable the text classification model to perform more-accurate classification based on text segments rather than (or in addition to) performing classification on single sentences. The classification of the text segments may be used to assign labels to the text segments to enable subsequent searches based on the labels.

The invention provides a free text-guided remote sensing image anaphora segmentation method and system, and the method comprises the steps: constructing a data sample which comprises an image, a text and various labels, inputting and training a region relation-driven image-text segmentation model, carrying out the multi-scale perception and dynamic response enhancement of the image through a dynamic correlation visual encoder of the model, and carrying out the real-time segmentation of the image. Generating a multi-scale visual feature # imgabs0 #; the pixel-level decoder is used for carrying out pixel-level decoding on the # imgabs 1 # and outputting image mask information # imgabs 2 #; the context association text encoder is used for carrying out semantic modeling on a text to generate attribute-object information # imgabs3 #; the regional relation modeling module is used for respectively carrying out regional-vision and regional-language modeling interaction on the # imgabs4 # and the # imgabs5 # to obtain a regional filter # imgabs6 # and a regional correlation feature # imgabs7 #; and the target guiding joint decoder is used for carrying out joint decoding on the # imgabs8 #, the # imgabs9 # and the # imgabs10 # so as to realize multi-head prediction output of the model. According to the invention, the remote sensing image can be segmented.

The invention discloses an RAG performance improvement method based on process optimization. The method comprises the following steps: S1, preprocessing text data; s2, performing hierarchical text segmentation, obtaining a document, storing the text by utilizing a database after the document is segmented, converting the text into vectors, and storing the vectors into a vector knowledge base; s3, querying and rewriting to obtain a rewritten question; s4, combining parent document search with mixed retrieval, and retrieving related contexts for the rewritten questions; s5, performing rearrangement and denoising to obtain a final context; and S6, generating an answer, and sending the final context into a large language model for understanding questions and answers. According to the method, the links of text preprocessing, text partitioning, text embedding, query optimization, retrieval, retrieval post-processing, generation and the like are sequentially optimized according to the data flowing direction, the semantic retrieval precision is improved, and the performance of RAG in the aspects of text retrieval precision, answer generation quality, user interaction and the like can be remarkably improved; and the flexibility and the practicability of the system are enhanced, so that the system is more excellent in practical application.

The invention provides a semantic enhancement adaptive partitioning method and system for natural resource large model questions and answers, and aims to solve the problems of difficulty in term boundary recognition, damage to semantic integrity and the like. According to the method, three core technologies including theme perception coarse-grained paragraph division, self-adaptive sliding window theme hierarchy division and embedded perception context self-adaptive text segmentation are fused. The system firstly analyzes a natural resource long text structure, identifies titles and theme levels and aligns associated contents; paragraphs are extracted according to a theme perception strategy and are subdivided into sentence sets according to grammar rules; an improved sliding window mechanism is adopted to divide sentences into window sentence block groups. The method is characterized in that a dynamic aggregation threshold mechanism is introduced, the semantic association degree between adjacent sentence blocks is calculated through an embedded perception context semantic segmentation technology, whether the sentence blocks are combined or not is judged by combining a similarity distribution change trend and a dynamic adjustment threshold, self-adaptive delimitation of semantic boundaries is achieved, and text blocks which are clear in structure and coherent in semantics are generated.

The invention relates to the technical field of artificial intelligence, and particularly discloses a policy and regulation intelligent question answering method and system based on a large language model, and the method comprises the steps: segmenting a policy and regulation text into text blocks through a recursive character text segmentation method, and constructing a knowledge base in combination with expert labeling and automatic information extraction; performing semantic embedding on the text block by utilizing a pre-trained Transform model, generating a high-dimensional vector representation, and storing the high-dimensional vector representation in a Chroma database; the user question is converted into an embedded vector, relevant laws and regulations are retrieved through a Chroma database, a large language model and a knowledge fusion attention generation module are combined, and text embedding and knowledge graph information are fused to generate an answer; and the user obtains answers through the intelligent question-answer interaction interface and can view the original texts and explanations of laws and regulations. The system comprises a data acquisition and preprocessing module, a semantic embedding and storage module, a retrieval and generation module and a user interaction module. According to the method, large-size regulation texts are effectively processed, and the retrieval efficiency and the answer quality are improved.

The invention discloses a marketing channel clustering modeling and optimizing method driven by heterogeneous data. The method comprises the steps of obtaining a multi-dimensional data set of a marketing channel, and performing preprocessing; the data set comprises structured data and unstructured data; and extracting a text word segmentation result and an emotion feature from the unstructured data, and extracting an image feature to generate a structured feature set, an unstructured feature set and the like. According to the method, structured and unstructured data can be integrated, and potential information of marketing channels can be comprehensively mined. Through a deep learning model (an auto-encoder, a variational auto-encoder and a generative adversarial network), features are extracted and optimized layer by layer, and the quality of the features and the performance of the model are improved. And in combination with a clustering algorithm and an evaluation index, the accuracy and reliability of a clustering result are ensured, and powerful support is provided for optimization of a marketing channel.

The invention discloses a method and system for generating a grade protection evaluation report based on RAG, and the method comprises the steps: carrying out the text segmentation of an input evaluation related file, and forming a plurality of text blocks; converting the text block into a vector form through a vector embedding model, and storing the text block into a semantic vector database; generating a corresponding user question according to terms in a preset evaluation template, and retrieving a target text block related to the user question from the semantic vector database by using a retrieval algorithm; and inputting the target text block into a pre-trained large language model, and generating evaluation contents corresponding to terms in combination with the keywords and terms in the evaluation template. The method does not depend on specific data formats or contents of specific companies, can flexibly cope with different conditions of different companies through the semantic vector database, the retrieval algorithm and the powerful natural language understanding and generation capability of the large model, automatically matches related terms and generates corresponding evaluation reports.

The invention relates to a professional knowledge graph construction method for the medical field, and the method comprises the following steps: obtaining a multi-modal medical document, carrying out the deep analysis through a Transform-based multi-modal model, and obtaining an analyzed multi-modal medical document; dynamically segmenting the analyzed multi-modal medical document by adopting a context-based adaptive text segmentation method to obtain independent text blocks; based on the independent text blocks, performing structured extraction, and performing verification by adopting a dynamic dual-verification mechanism to form knowledge sub-graphs; and based on the knowledge sub-graph, further constructing a standardized medical knowledge graph. Compared with the prior art, the method has the advantages of high precision, expandability and the like.

The invention provides an intelligent research and judgment method and system for network-related security check terms of a network source based on a large language model. The method comprises the following steps: S1, collecting network-related security evaluation terms and a basis file; s2, performing pre-training and fine tuning on the universal Embedding model by using a network-related security evaluation clause and a basis file so as to construct a network source special model; s3, carrying out preprocessing and text segmentation on the network-related security evaluation basis file, carrying out text vectorization processing through a network source special model, and storing vectorized text fragments and corresponding original texts into a database to construct a network source security special knowledge base; and S4, carrying out intelligent research and judgment on network-related security check terms by combining the large model and a network source security special knowledge base. According to the method, the advantages of the large language model in the aspect of natural language understanding are fully played, and the clause basis file is accurately positioned in combination with the network source security special model, so that a large number of unstructured network-related security check clauses can be efficiently processed, and intelligent and efficient research and judgment are realized.

The invention relates to the field of artificial intelligence, in particular to a long text segmentation information retrieval result coherence enhancement method, system and equipment based on a knowledge graph, and the method comprises the following steps: receiving a long text input by a user; segmenting the long text into a plurality of segments according to a preset length, and recording the original position and sequence of each segment; performing named entity recognition on each fragment, and extracting a key entity; querying related information in a knowledge graph according to the key entity, and obtaining attributes of the entity and a relationship between the attributes; integrating the related information obtained by query into the corresponding text fragment to form a context enhanced fragment; receiving a user query request; calculating a correlation score based on the context enhanced fragment and the user query, and retrieving a related text fragment by using a hybrid retrieval strategy; and selecting the related text fragment with the highest score to generate a final answer, and outputting the final answer to the user. Therefore, the information integrity and the context continuity in the retrieval process are ensured.

The invention discloses a global knowledge extraction method and system based on a large model and an RAG technology, and belongs to the technical field of large model data processing. According to the global knowledge extraction method and system based on the large model and the RAG technology, local sensitive hashing is adopted for duplicate removal, similar texts are combined through dynamic thresholds, and redundant data interference is reduced; a text segmentation strategy based on separator confidence ensures that segmented text blocks maintain semantic integrity and are adaptive to large model input length limitation, similarity retrieval and full-text retrieval are fused, results are reordered in combination with an RRF algorithm, semantic relevance and accuracy are considered, the multi-scene knowledge construction requirement is met, and the multi-scene knowledge construction efficiency is improved. High-correlation entity pairs are screened through mutual information, context knowledge is retrieved from a vector database in combination with an RAG system, semantic reasoning is conducted through a large language model, hidden logic relations among entities are generated, and jumping from data association to knowledge generation is achieved.

The invention provides a text segmentation method and related equipment. The method comprises the steps of obtaining a to-be-processed text; segmenting the to-be-processed text into ordered statement sequences to obtain an initial statement set of the to-be-processed text; wherein the ordered statement sequence comprises a plurality of statements; the semantic variation, the confusion degree variation and the information entropy variation of a first target text block are calculated when a to-be-decided statement in an initial statement set of the to-be-processed text is added into the first target text block, and the first target text block is a set of multiple statements meeting a merging condition; determining the collaboration degree of the statement to be decided and the first target text block based on the semantic variable quantity, the confusion variable quantity and the information entropy variable quantity; and partitioning the to-be-processed text based on the collaboration degree of the to-be-decided statement and the first target text block to obtain a target partitioned text of the to-be-processed text. The text segmentation quality can be improved.

The invention provides a civil engineering field knowledge large model retrieval enhancement generation method. The method comprises civil engineering domain knowledge processing and vectorization storage, civil engineering domain knowledge capability evaluation and adaptation of a large language model, a civil engineering domain knowledge vector database efficient retrieval mechanism and knowledge enhancement generation guidance based on retrieval results. According to the method, knowledge in the civil engineering field is encoded and established into a vector database through preprocessing processes such as data cleaning and text segmentation; a general large language model suitable for civil engineering field knowledge retrieval enhancement generation is selected through a semantic confusion index, target knowledge is retrieved through similarity calculation to form cue words, and the civil engineering field professional knowledge generation ability of the large language model is enhanced.

The invention provides a knowledge segmentation retrieval system based on nltk, and belongs to the technical field of text segmentation and retrieval, and the system comprises a knowledge uploading and segmenting module which is used for receiving and analyzing docx, pdf and txt files, segmenting the files into knowledge segments, and establishing an index; the knowledge retrieval module is used for receiving a query request of a user, quickly positioning the knowledge fragment with the highest similarity through the constructed index, and returning a retrieval result; the knowledge slice paragraph query module is used for providing a paragraph content query service after segmentation and carrying out sorting and returning according to paragraph serial numbers; and the knowledge deleting module is used for deleting a single knowledge file or the whole knowledge base and ensuring the consistency and integrity of the data. According to the method, accurate segmentation and efficient retrieval of long texts in the knowledge base can be realized, the knowledge management efficiency and quality are greatly improved, and convenient and accurate knowledge services are provided for users.

The invention relates to the technical field of natural language processing, in particular to an RAG retrieval process optimization method and system based on a fine-tuning embedding model, and the method comprises the steps: determining the topic expression distribution degree of each topic in each long text document; obtaining all window sequences of each text segmentation topic based on the topic intensity sequence of each text segmentation topic by adopting a sliding window algorithm, and determining the topic mutation degree of each window sequence of each text segmentation topic based on the difference of the average distribution condition between the first half element and the second half element in each window sequence, short texts under all text segmentation themes are obtained; and determining the text topic concentration ratio of each short text, and optimizing the RAG retrieval process in combination with a large language model. According to the method, the text segmentation quality is improved by analyzing the topic expression distribution conditions of different topics in the long text document.

The invention provides a document data processing system and method based on a lightweight model, and the system comprises a document conversion module which is used for converting a to-be-processed document into a picture file; the layout analysis module is used for processing the picture file according to a pre-constructed layout analysis lightweight model, and extracting the category and area coordinates of each label in the picture file; the layout restoration module is used for respectively analyzing and processing different types of label areas according to the type and the area coordinate of each label to obtain the content corresponding to each label area; obtaining all contents of the to-be-processed document by combining the contents corresponding to each label area; and the text segmentation module is used for segmenting all contents of the to-be-processed document into text blocks with preset sizes through a divider, and completing format conversion and document data processing of the to-be-processed document. And rapid identification and separation of graphs, tables and characters in the PDF format document are realized.

The invention relates to the technical field, in particular to an AI classroom teaching quality evaluation method and system based on image recognition, and the method comprises the steps: obtaining a test paper RGB image and structured metadata, and generating multi-modal input data through graying, denoising and affine transformation; dynamically adjusting the weight through a double-branch feature extraction network (CNN + ViT), and extracting the global structure of the printed form and the local detail features of the handwritten form; generating a mixed font segmentation mask by using space attention, channel attention and dynamic routing regulation; optimizing handwritten answer recognition in combination with an educational knowledge graph; and fusing the answer expression and the attention data to generate a teaching quality evaluation report. The system comprises a data acquisition unit, a preprocessing unit, a feature extraction unit, a segmentation unit, an identification verification unit and an evaluation unit. According to the scheme, the accuracy of character segmentation and recognition in a mixed font scene is improved, and data support is provided for precise teaching in an education scene.

The invention belongs to the technical field of text-to-speech conversion, and particularly relates to a solution for TTS (text-to-speech) in a high-concurrency scene through tone cloning. According to the method, the text can be generated and returned to the client by segmenting and parallelizing the text and reducing the delay of the first segment, compared with a traditional serial whole segment synthesis mode, the waiting time of a user is greatly shortened, the audio is returned after being generated, the waiting time of the user can be effectively shortened under high concurrency, the network bandwidth and the delay pressure are reduced, and the user experience is improved. The real-time performance is guaranteed, reconnection disasters are avoided, underlying model services can be fully utilized, the influence of the serial processing characteristic of a CosyVoice2 model is avoided, a client can immediately start to play a first segment of audio, even if subsequent segments are still synthesized, the continuous playing experience of previous content is not influenced, and when the concurrency is increased, the continuous playing experience of the previous content is not influenced. The model gateway can dynamically disperse requests to a plurality of GPU instances, and long queue waiting after a single-card video memory is fully occupied is avoided.

The invention discloses a text segmentation method, a question answering method and related devices and equipments.The text segmentation method comprises the steps that the confusion degree of each sentence text in a to-be-processed text is obtained; for each sentence text, obtaining a confusion degree comparison result between the sentence text and the adjacent sentence text, and determining whether to select the sentence text as a segmentation point of the to-be-processed chapter based on the comparison result; and segmenting the to-be-processed chapter based on each segmentation point to obtain a plurality of text blocks. According to the scheme, the segmentation adaptivity of the chapter text can be improved, and the semantic and logic coherence of the text blocks obtained through segmentation can be improved.

The invention provides a document retrieval enhancement method, device and equipment for a large language model. The method comprises the steps that for each original text, the starting position of the original text serves as a starting point, a text segmentation symbol is searched for according to the text sequence of the text, text segmentation is conducted on the basis of a preset heuristic rule and a sliding window according to the starting point and the text segmentation symbol, and a plurality of text blocks meeting the text length range are obtained; obtaining all word sizes in the original text, and generating a document directory structure corresponding to the original text; storing the document directory structure corresponding to the original text and the vectorized text fragments of the corresponding text blocks into a vector database; recalling topK recalled text fragments most related to user input; and for each recalled text fragment, according to the corresponding document directory structure, obtaining a vectorized text fragment corresponding to the same level or higher level of the recalled text fragment, and obtaining a recalled long text meeting the length range of the recalled text.

The invention relates to the technical field of information extraction, and discloses a long text news event element extraction method and system based on multi-agent collaboration, and the method comprises the following steps: constructing a large language model multi-agent system architecture; the central coordination agent receives the unstructured news text; the text segmentation agent outputs a logic unit and time description thereof; an event extraction agent extracts an event subject, an action and an object, and complements time and entity role information; the scene matching agent judges the fine-grained news scene category to which the scene matching agent belongs; the information extraction agent generates a structured event record; and the central coordination agent performs quality auditing and exception handling on each step of output to ensure the integrity and consistency of the output data. According to the method, the static relationship between the event main bodies can be constructed according to the structured event information through the relationship construction and context sorting agent, the event situation change is summarized based on the timeline, and the feasibility and the visualization of news event analysis of the system are improved.

The invention relates to an intelligent text segmentation method and system based on a large language model, and the method is characterized in that the method comprises the following steps: receiving original text data, cleaning and normalizing the data, and obtaining a preprocessed text; performing deep semantic analysis on the preprocessed text by using a pre-trained large language model; identifying important semantic units in the preprocessed text, and performing text segmentation according to semantic boundaries to obtain text segments; the segmented text segments are converted into vector representation; and the generated vectors are used for semantic matching, and a generation or retrieval task is executed, so that an intelligent word segmentation result better fits the overall semantics and context of a text, and the correlation of word segmentation is effectively improved.

The invention discloses a supply chain fraud behavior early warning method and device based on a large language model, and relates to the field of data analysis, and the method comprises the steps: obtaining supply chain multi-source data, and carrying out the processing of the supply chain multi-source data according to the data type; in the fine tuning process, a LoRA module is injected into a linear layer of a pre-trained large language model base, and the rank value of the LoRA module is dynamically adjusted according to the gradient; according to the text word segmentation data, the entity type, the aligned historical order data, the aligned historical logistics data and the dynamic space-time diagram, constructing a multi-modal prefix guide vector; the multi-modal prefix guide vector and an original input sequence of a linear layer of a pre-trained large language model base are spliced and then input into the linear layer of the pre-trained large language model base, a supply chain fraud behavior early warning model subjected to fine adjustment is obtained through fine adjustment, and early warning is conducted on supply chain fraud behaviors of related suppliers. According to the method, the problems of low supply chain fraud behavior identification accuracy, large training parameters and the like in the prior art are solved.

The invention provides a text segmentation method and device and terminal equipment. The method comprises the steps of obtaining a text and generating duration of the text; based on the duration, determining a text segmentation strategy associated with the text; based on the text segmentation strategy, segmenting the text to obtain a first sentence text of the text; and performing text-to-speech processing on the first sentence text to obtain speech corresponding to the first sentence text.

The invention belongs to the field of artificial intelligence technology machine learning and semantic understanding, and relates to semantic analysis and semantic recognition, in particular to a text block semantic coherence detection method oriented to an RAG question and answer system, which specifically quantifies semantic loss of different blocks, assists in rejecting sampling to construct a supervised fine tuning (SFT) and reinforcement learning (RL) training set, and improves the semantic coherence of the different blocks. Therefore, the text partitioning capability of the large language model is improved, and the retrieval effect and the answer generation quality of the RAG system are optimized. According to the method, the semantic consistency in the blocks and the semantic jump degree at the boundaries of the blocks are quantified, an objective basis is provided for evaluating the quality of the text blocks, the effectiveness of the algorithm is verified through design experiments, and the method has important significance in improving the text blocking capacity of a large language model, the retrieval effect of an RAG question and answer system and the answer generation quality.

The embodiment of the invention provides a text field detection method and device based on text segmentation and hierarchical abstracts.The method comprises the steps that a target text is segmented to obtain a plurality of text segments, and therefore the text length can be shortened and parallelization processing can be conducted; the tree structure and the hierarchical abstract are formed through the text segments, so that the target text can be deeply understood from multiple dimensions; according to the abstract text of the child node, the abstract text of the middle node and the abstract text of the root node, obtaining a text field of the text fragment; determining the text field of the abstract text of the root node according to the text field of the text fragment, the similarity of the abstract text of the child node and the similarity of the abstract text of the intermediate node, and taking the text field of the abstract text of the root node as the text field of the target text, so that more comprehensive information of the text can be obtained, and the user experience is improved. The method greatly improves the detection capability of the text field, is transparent in analysis process, provides better interpretability, and is convenient for a user to deeply understand.

The embodiment of the invention provides a cultural resource digitization implementation method based on image recognition, and the method comprises the steps: extracting a text candidate region in an enhanced image based on text edge positioning and non-text region filtering, carrying out the text segmentation of the text candidate region through employing a DB segmentation algorithm, carrying out the segmentation processing of a text line image obtained through segmentation, and carrying out the segmentation processing of the text line image. A coincident text segment exists between the segment tail of the first text segment and the segment head of the second text segment; performing text recognition on the first text segment and the second text segment by using a text recognition model, and using an obtained recognition result of the coincident text segment for optimization adjustment of the text recognition model; and taking a new to-be-recognized image as input of the optimized and adjusted text recognition model to obtain a text recognition result of the current to-be-recognized image, and constructing and outputting a culture resource database containing the text recognition result. According to the embodiment of the invention, image information such as ancient books and literatures can be digitalized, and a culture resource database convenient to manage and apply is constructed.

The invention provides a hospitalization first-time disease course record generation method and system based on a large model, and the method comprises the steps: carrying out the text segmentation, format conversion, term standardization and theme recognition through employing a historical hospitalization document record, obtaining semantic transcription data through the similarity, building a mapping relation, and carrying out the calculation of the semantic transcription data. According to the method, the medical record documents called and stored in the electronic medical record system are processed, semantic transcription data corresponding to the medical record documents are obtained, and case medical records with similarity meeting a threshold value and stored in the electronic medical record system are matched through keyword and semantic mixed retrieval according to the medical record documents and the semantic transcription data. According to the semantic transcription data and the first course record initial cue word, constructing an optimized cue word with department and disease type characteristics, and according to the optimized cue word, the semantic transcription data and the case medical record, inputting the optimized cue word, the semantic transcription data and the case medical record into a hospitalization first course record generation model trained by adopting historical data to finally obtain a hospitalization first course record. The problem that in the prior art, the accuracy of the first record content is low is solved.