398 results about "Text segmentation" patented technology

The invention discloses a grassroots power supply enterprise compliance risk intelligent system and method based on big data analysis. The data acquisition unit is used for acquiring business operation data, historical violation records and policy and regulation update data of basic power supply enterprises to form a unified compliance data set. And the natural language processing unit performs text word segmentation and correlation analysis on the policy and regulation and violation record data, extracts key risk factors and labels compliance risk labels. And the risk feature construction unit performs multi-dimensional feature fusion on the business operation data and the compliance risk label data to generate a feature matrix for risk identification. And the intelligent risk assessment unit performs real-time analysis on the feature matrix by using a pre-trained machine learning model, identifies compliance risk categories and levels, and generates early warning information. According to the method, the accuracy of compliance risk identification is improved by using big data analysis and an intelligent algorithm, the compliance management cost of basic-level power supply enterprises is reduced, and the operation safety and compliance of the enterprises are improved.

The present application relates to a retrieval model training method and apparatus and a computer device. The method comprises: acquiring a knowledge document, and performing text segmentation processing on the knowledge document to obtain a text block set (501); on the basis of the text block set and summary information comprised in each text block, writing a question for each text block to obtain one or more question texts corresponding to each text block, separately combining the one or more question texts corresponding to each text block with the corresponding text block to obtain one or more data pairs corresponding to each text block and, on the basis of the one or more data pairs corresponding to each text block, generating a data pair set (502); and using the data pair set to train a first retrieval model so as to obtain a second retrieval model (503).

Systems and methods for semantic temporal segmentation based on topic recognition are disclosed. Text classification and segmentation may be used to index media content for subsequent searches. The method may include using a text classification model to semantically analyze sentences in a text file (such as a transcript) to determine topics with which sentences are associated. The output of the text classification model may be provided to a text segmentation model to enable more-accurate identification of text segments (e.g., paragraphs) within the text file. In some examples, the output of a text segmentation model is provided to a text classification model to enable the text classification model to perform more-accurate classification based on text segments rather than (or in addition to) performing classification on single sentences. The classification of the text segments may be used to assign labels to the text segments to enable subsequent searches based on the labels.

The invention provides a free text-guided remote sensing image anaphora segmentation method and system, and the method comprises the steps: constructing a data sample which comprises an image, a text and various labels, inputting and training a region relation-driven image-text segmentation model, carrying out the multi-scale perception and dynamic response enhancement of the image through a dynamic correlation visual encoder of the model, and carrying out the real-time segmentation of the image. Generating a multi-scale visual feature # imgabs0 #; the pixel-level decoder is used for carrying out pixel-level decoding on the # imgabs 1 # and outputting image mask information # imgabs 2 #; the context association text encoder is used for carrying out semantic modeling on a text to generate attribute-object information # imgabs3 #; the regional relation modeling module is used for respectively carrying out regional-vision and regional-language modeling interaction on the # imgabs4 # and the # imgabs5 # to obtain a regional filter # imgabs6 # and a regional correlation feature # imgabs7 #; and the target guiding joint decoder is used for carrying out joint decoding on the # imgabs8 #, the # imgabs9 # and the # imgabs10 # so as to realize multi-head prediction output of the model. According to the invention, the remote sensing image can be segmented.

The invention discloses an RAG performance improvement method based on process optimization. The method comprises the following steps: S1, preprocessing text data; s2, performing hierarchical text segmentation, obtaining a document, storing the text by utilizing a database after the document is segmented, converting the text into vectors, and storing the vectors into a vector knowledge base; s3, querying and rewriting to obtain a rewritten question; s4, combining parent document search with mixed retrieval, and retrieving related contexts for the rewritten questions; s5, performing rearrangement and denoising to obtain a final context; and S6, generating an answer, and sending the final context into a large language model for understanding questions and answers. According to the method, the links of text preprocessing, text partitioning, text embedding, query optimization, retrieval, retrieval post-processing, generation and the like are sequentially optimized according to the data flowing direction, the semantic retrieval precision is improved, and the performance of RAG in the aspects of text retrieval precision, answer generation quality, user interaction and the like can be remarkably improved; and the flexibility and the practicability of the system are enhanced, so that the system is more excellent in practical application.

The invention provides a semantic enhancement adaptive partitioning method and system for natural resource large model questions and answers, and aims to solve the problems of difficulty in term boundary recognition, damage to semantic integrity and the like. According to the method, three core technologies including theme perception coarse-grained paragraph division, self-adaptive sliding window theme hierarchy division and embedded perception context self-adaptive text segmentation are fused. The system firstly analyzes a natural resource long text structure, identifies titles and theme levels and aligns associated contents; paragraphs are extracted according to a theme perception strategy and are subdivided into sentence sets according to grammar rules; an improved sliding window mechanism is adopted to divide sentences into window sentence block groups. The method is characterized in that a dynamic aggregation threshold mechanism is introduced, the semantic association degree between adjacent sentence blocks is calculated through an embedded perception context semantic segmentation technology, whether the sentence blocks are combined or not is judged by combining a similarity distribution change trend and a dynamic adjustment threshold, self-adaptive delimitation of semantic boundaries is achieved, and text blocks which are clear in structure and coherent in semantics are generated.

The invention discloses a software configuration item test case automatic generation method based on a large language model, and relates to the technical field of software testing. In order to solve the problems that in the prior art, the flexibility and transportability of a system are limited, the system is difficult to adapt to constantly changing software requirements and designs, and manual intervention is still needed to generate and analyze a test sequence. According to the automatic generation method for the software configuration item test case based on the large language model, through intelligent text segmentation, prompt word construction and test case generation, the efficiency and accuracy of test case generation are improved, the pertinence and coverage of the test case are ensured, and the workload of manually compiling the test case is remarkably reduced; a knowledge graph of software configuration items is constructed, so that the understanding of the model on correlation and similarity among texts is enhanced, and the quality of test case generation is improved; and through a real-time feedback and optimization mechanism, the automation level and the test efficiency of software testing are effectively improved.

The invention discloses an intelligent construction and retrieval method for a medical knowledge graph based on GraphRAG and LLM. The intelligent construction method of the medical knowledge graph comprises the steps of data collection and processing, text segmentation, entity and relation extraction, entity analysis, weak connected component identification, word distance filtering, LLM evaluation, element abstract generation, graph community generation and community abstract generation. The intelligent retrieval method comprises the steps of knowledge graph vector library construction, text vector library construction, user question entity extraction and vectorization processing, multi-layer entity and relation retrieval, text retrieval, LLM summarization, result output and user feedback collection. According to the method, the GraphRAG and the LLM large model technology are combined, and efficient retrieval and accurate presentation of the medical information are achieved. According to the system, the accuracy and efficiency of medical literature retrieval are remarkably improved through deep integration of graph structured data and natural language processing capacity.

The invention relates to the technical field of artificial intelligence, and particularly discloses a policy and regulation intelligent question answering method and system based on a large language model, and the method comprises the steps: segmenting a policy and regulation text into text blocks through a recursive character text segmentation method, and constructing a knowledge base in combination with expert labeling and automatic information extraction; performing semantic embedding on the text block by utilizing a pre-trained Transform model, generating a high-dimensional vector representation, and storing the high-dimensional vector representation in a Chroma database; the user question is converted into an embedded vector, relevant laws and regulations are retrieved through a Chroma database, a large language model and a knowledge fusion attention generation module are combined, and text embedding and knowledge graph information are fused to generate an answer; and the user obtains answers through the intelligent question-answer interaction interface and can view the original texts and explanations of laws and regulations. The system comprises a data acquisition and preprocessing module, a semantic embedding and storage module, a retrieval and generation module and a user interaction module. According to the method, large-size regulation texts are effectively processed, and the retrieval efficiency and the answer quality are improved.

The invention discloses a marketing channel clustering modeling and optimizing method driven by heterogeneous data. The method comprises the steps of obtaining a multi-dimensional data set of a marketing channel, and performing preprocessing; the data set comprises structured data and unstructured data; and extracting a text word segmentation result and an emotion feature from the unstructured data, and extracting an image feature to generate a structured feature set, an unstructured feature set and the like. According to the method, structured and unstructured data can be integrated, and potential information of marketing channels can be comprehensively mined. Through a deep learning model (an auto-encoder, a variational auto-encoder and a generative adversarial network), features are extracted and optimized layer by layer, and the quality of the features and the performance of the model are improved. And in combination with a clustering algorithm and an evaluation index, the accuracy and reliability of a clustering result are ensured, and powerful support is provided for optimization of a marketing channel.

The invention discloses a method and system for generating a grade protection evaluation report based on RAG, and the method comprises the steps: carrying out the text segmentation of an input evaluation related file, and forming a plurality of text blocks; converting the text block into a vector form through a vector embedding model, and storing the text block into a semantic vector database; generating a corresponding user question according to terms in a preset evaluation template, and retrieving a target text block related to the user question from the semantic vector database by using a retrieval algorithm; and inputting the target text block into a pre-trained large language model, and generating evaluation contents corresponding to terms in combination with the keywords and terms in the evaluation template. The method does not depend on specific data formats or contents of specific companies, can flexibly cope with different conditions of different companies through the semantic vector database, the retrieval algorithm and the powerful natural language understanding and generation capability of the large model, automatically matches related terms and generates corresponding evaluation reports.

The invention relates to a professional knowledge graph construction method for the medical field, and the method comprises the following steps: obtaining a multi-modal medical document, carrying out the deep analysis through a Transform-based multi-modal model, and obtaining an analyzed multi-modal medical document; dynamically segmenting the analyzed multi-modal medical document by adopting a context-based adaptive text segmentation method to obtain independent text blocks; based on the independent text blocks, performing structured extraction, and performing verification by adopting a dynamic dual-verification mechanism to form knowledge sub-graphs; and based on the knowledge sub-graph, further constructing a standardized medical knowledge graph. Compared with the prior art, the method has the advantages of high precision, expandability and the like.

The invention provides an intelligent research and judgment method and system for network-related security check terms of a network source based on a large language model. The method comprises the following steps: S1, collecting network-related security evaluation terms and a basis file; s2, performing pre-training and fine tuning on the universal Embedding model by using a network-related security evaluation clause and a basis file so as to construct a network source special model; s3, carrying out preprocessing and text segmentation on the network-related security evaluation basis file, carrying out text vectorization processing through a network source special model, and storing vectorized text fragments and corresponding original texts into a database to construct a network source security special knowledge base; and S4, carrying out intelligent research and judgment on network-related security check terms by combining the large model and a network source security special knowledge base. According to the method, the advantages of the large language model in the aspect of natural language understanding are fully played, and the clause basis file is accurately positioned in combination with the network source security special model, so that a large number of unstructured network-related security check clauses can be efficiently processed, and intelligent and efficient research and judgment are realized.

The invention discloses a text segmentation method and device for retrieval enhancement generation, and the method comprises the steps: constructing a text tree for an original text, and enabling the text tree to take an original text attribute as a root node, a title of each level as a child node, and a non-title paragraph as a leaf node; according to the method, text blocks are segmented according to a text tree, so that each segmented text block starts from a directory structure, ends with a non-title paragraph and is constrained by preset text block word number limitation, and the directory structure included in each text block is as follows: starting from a head leaf node included in each text block, starting from a non-title paragraph included in each text block, and ending with a non-title paragraph included in each text block; according to the method and the device, the text can be segmented in a mode more suitable for a tree structure, and text information loss caused in the text segmentation process is reduced, so that the recall effect of an RAG system is improved.

The invention discloses a question and answer generation method and device, equipment and a storage medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining a text, and carrying out the text segmentation of the text to obtain a preset number of text blocks; performing vector retrieval on the text block to obtain a vector result; performing knowledge graph retrieval on the text block to obtain a knowledge graph retrieval result; target answer information is generated according to a vector result and a knowledge graph retrieval result, a mixed retrieval mode of vector retrieval and knowledge graph retrieval is used, the advantages of structured graph reasoning and unstructured semantic retrieval are fused, efficient response of complex query and dynamic updating of a knowledge graph are achieved, the knowledge coverage range is remarkably widened, and the search efficiency is improved. Meanwhile, the big language model reasoning cost is reduced, and the information retrieval and generation speed is increased.

The invention relates to the field of artificial intelligence, in particular to a long text segmentation information retrieval result coherence enhancement method, system and equipment based on a knowledge graph, and the method comprises the following steps: receiving a long text input by a user; segmenting the long text into a plurality of segments according to a preset length, and recording the original position and sequence of each segment; performing named entity recognition on each fragment, and extracting a key entity; querying related information in a knowledge graph according to the key entity, and obtaining attributes of the entity and a relationship between the attributes; integrating the related information obtained by query into the corresponding text fragment to form a context enhanced fragment; receiving a user query request; calculating a correlation score based on the context enhanced fragment and the user query, and retrieving a related text fragment by using a hybrid retrieval strategy; and selecting the related text fragment with the highest score to generate a final answer, and outputting the final answer to the user. Therefore, the information integrity and the context continuity in the retrieval process are ensured.

Data such as unstructured text is received that includes a sequence of sentences. This received data is then tokenized into a plurality of tokens. The received data is segmented using a hierarchical transformer network model including a token transformer, a sentence transformer, and a segmentation classifier. The token transformer contextualizes tokens within sentences and yields sentence embeddings. The sentences transformer contextualizes sentence representations based on the sentence embedddings. The segmentation classifier predicts segments of the received data based on the contextualized sentence representations. Data can be provided which characterizes the segmentation of the received data. Related apparatus, systems, techniques and articles are also described.

The invention discloses a global knowledge extraction method and system based on a large model and an RAG technology, and belongs to the technical field of large model data processing. According to the global knowledge extraction method and system based on the large model and the RAG technology, local sensitive hashing is adopted for duplicate removal, similar texts are combined through dynamic thresholds, and redundant data interference is reduced; a text segmentation strategy based on separator confidence ensures that segmented text blocks maintain semantic integrity and are adaptive to large model input length limitation, similarity retrieval and full-text retrieval are fused, results are reordered in combination with an RRF algorithm, semantic relevance and accuracy are considered, the multi-scene knowledge construction requirement is met, and the multi-scene knowledge construction efficiency is improved. High-correlation entity pairs are screened through mutual information, context knowledge is retrieved from a vector database in combination with an RAG system, semantic reasoning is conducted through a large language model, hidden logic relations among entities are generated, and jumping from data association to knowledge generation is achieved.

The invention provides a text segmentation method and related equipment. The method comprises the steps of obtaining a to-be-processed text; segmenting the to-be-processed text into ordered statement sequences to obtain an initial statement set of the to-be-processed text; wherein the ordered statement sequence comprises a plurality of statements; the semantic variation, the confusion degree variation and the information entropy variation of a first target text block are calculated when a to-be-decided statement in an initial statement set of the to-be-processed text is added into the first target text block, and the first target text block is a set of multiple statements meeting a merging condition; determining the collaboration degree of the statement to be decided and the first target text block based on the semantic variable quantity, the confusion variable quantity and the information entropy variable quantity; and partitioning the to-be-processed text based on the collaboration degree of the to-be-decided statement and the first target text block to obtain a target partitioned text of the to-be-processed text. The text segmentation quality can be improved.

The invention provides a civil engineering field knowledge large model retrieval enhancement generation method. The method comprises civil engineering domain knowledge processing and vectorization storage, civil engineering domain knowledge capability evaluation and adaptation of a large language model, a civil engineering domain knowledge vector database efficient retrieval mechanism and knowledge enhancement generation guidance based on retrieval results. According to the method, knowledge in the civil engineering field is encoded and established into a vector database through preprocessing processes such as data cleaning and text segmentation; a general large language model suitable for civil engineering field knowledge retrieval enhancement generation is selected through a semantic confusion index, target knowledge is retrieved through similarity calculation to form cue words, and the civil engineering field professional knowledge generation ability of the large language model is enhanced.

The invention provides a knowledge segmentation retrieval system based on nltk, and belongs to the technical field of text segmentation and retrieval, and the system comprises a knowledge uploading and segmenting module which is used for receiving and analyzing docx, pdf and txt files, segmenting the files into knowledge segments, and establishing an index; the knowledge retrieval module is used for receiving a query request of a user, quickly positioning the knowledge fragment with the highest similarity through the constructed index, and returning a retrieval result; the knowledge slice paragraph query module is used for providing a paragraph content query service after segmentation and carrying out sorting and returning according to paragraph serial numbers; and the knowledge deleting module is used for deleting a single knowledge file or the whole knowledge base and ensuring the consistency and integrity of the data. According to the method, accurate segmentation and efficient retrieval of long texts in the knowledge base can be realized, the knowledge management efficiency and quality are greatly improved, and convenient and accurate knowledge services are provided for users.

The invention relates to the technical field of natural language processing, in particular to an RAG retrieval process optimization method and system based on a fine-tuning embedding model, and the method comprises the steps: determining the topic expression distribution degree of each topic in each long text document; obtaining all window sequences of each text segmentation topic based on the topic intensity sequence of each text segmentation topic by adopting a sliding window algorithm, and determining the topic mutation degree of each window sequence of each text segmentation topic based on the difference of the average distribution condition between the first half element and the second half element in each window sequence, short texts under all text segmentation themes are obtained; and determining the text topic concentration ratio of each short text, and optimizing the RAG retrieval process in combination with a large language model. According to the method, the text segmentation quality is improved by analyzing the topic expression distribution conditions of different topics in the long text document.

The invention provides a knowledge graph construction method and system based on LLM multi-task iterative prompt. The method comprises the steps of obtaining an industry document to be subjected to entity relationship extraction; performing entity relationship extraction on the industry document through a fine-tuned large language model in combination with preset prompt information to obtain extracted entities and relationships; wherein the extraction task of the entity relationship is divided into a text segmentation sub-task, an entity marking sub-task, an entity relationship extraction sub-task, an implicit relationship prediction sub-task and an entity relationship semantic merging sub-task, and each sub-task is realized through a fine-tuned large language model; each subtask is iteratively executed for a preset number of times to obtain a task result; and performing triple construction based on the obtained entities and relationships to obtain the industry knowledge graph.

The invention provides a document data processing system and method based on a lightweight model, and the system comprises a document conversion module which is used for converting a to-be-processed document into a picture file; the layout analysis module is used for processing the picture file according to a pre-constructed layout analysis lightweight model, and extracting the category and area coordinates of each label in the picture file; the layout restoration module is used for respectively analyzing and processing different types of label areas according to the type and the area coordinate of each label to obtain the content corresponding to each label area; obtaining all contents of the to-be-processed document by combining the contents corresponding to each label area; and the text segmentation module is used for segmenting all contents of the to-be-processed document into text blocks with preset sizes through a divider, and completing format conversion and document data processing of the to-be-processed document. And rapid identification and separation of graphs, tables and characters in the PDF format document are realized.

The invention relates to the technical field, in particular to an AI classroom teaching quality evaluation method and system based on image recognition, and the method comprises the steps: obtaining a test paper RGB image and structured metadata, and generating multi-modal input data through graying, denoising and affine transformation; dynamically adjusting the weight through a double-branch feature extraction network (CNN + ViT), and extracting the global structure of the printed form and the local detail features of the handwritten form; generating a mixed font segmentation mask by using space attention, channel attention and dynamic routing regulation; optimizing handwritten answer recognition in combination with an educational knowledge graph; and fusing the answer expression and the attention data to generate a teaching quality evaluation report. The system comprises a data acquisition unit, a preprocessing unit, a feature extraction unit, a segmentation unit, an identification verification unit and an evaluation unit. According to the scheme, the accuracy of character segmentation and recognition in a mixed font scene is improved, and data support is provided for precise teaching in an education scene.

The invention relates to the technical field of natural language processing, and discloses an Internet finance new entity recognition method and device based on big data, and the method comprises the steps: segmenting obtained text comment data, and obtaining a text segmentation data set; performing semantic processing on each piece of text segmentation data in the text segmentation data set to obtain a global text semantic vector set; inputting each global text semantic vector in the global text semantic vector set into a module based on a bidirectional entity boundary recognition neural network structure for expression analysis to obtain a text distributed expression tensor set; performing entity boundary classification on each text in the text distributed expression tensor set in a distributed manner to obtain a text head and tail entity boundary set; and performing entity head-tail merging processing on each text head-tail entity boundary in the text head-tail entity boundary set through a greedy matching algorithm to obtain a target entity corresponding to the text comment data, so that the target entity in the text comment data can be accurately extracted.

The invention belongs to the technical field of text-to-speech conversion, and particularly relates to a solution for TTS (text-to-speech) in a high-concurrency scene through tone cloning. According to the method, the text can be generated and returned to the client by segmenting and parallelizing the text and reducing the delay of the first segment, compared with a traditional serial whole segment synthesis mode, the waiting time of a user is greatly shortened, the audio is returned after being generated, the waiting time of the user can be effectively shortened under high concurrency, the network bandwidth and the delay pressure are reduced, and the user experience is improved. The real-time performance is guaranteed, reconnection disasters are avoided, underlying model services can be fully utilized, the influence of the serial processing characteristic of a CosyVoice2 model is avoided, a client can immediately start to play a first segment of audio, even if subsequent segments are still synthesized, the continuous playing experience of previous content is not influenced, and when the concurrency is increased, the continuous playing experience of the previous content is not influenced. The model gateway can dynamically disperse requests to a plurality of GPU instances, and long queue waiting after a single-card video memory is fully occupied is avoided.

The invention discloses a text segmentation method, a question answering method and related devices and equipments.The text segmentation method comprises the steps that the confusion degree of each sentence text in a to-be-processed text is obtained; for each sentence text, obtaining a confusion degree comparison result between the sentence text and the adjacent sentence text, and determining whether to select the sentence text as a segmentation point of the to-be-processed chapter based on the comparison result; and segmenting the to-be-processed chapter based on each segmentation point to obtain a plurality of text blocks. According to the scheme, the segmentation adaptivity of the chapter text can be improved, and the semantic and logic coherence of the text blocks obtained through segmentation can be improved.

The invention provides a document retrieval enhancement method, device and equipment for a large language model. The method comprises the steps that for each original text, the starting position of the original text serves as a starting point, a text segmentation symbol is searched for according to the text sequence of the text, text segmentation is conducted on the basis of a preset heuristic rule and a sliding window according to the starting point and the text segmentation symbol, and a plurality of text blocks meeting the text length range are obtained; obtaining all word sizes in the original text, and generating a document directory structure corresponding to the original text; storing the document directory structure corresponding to the original text and the vectorized text fragments of the corresponding text blocks into a vector database; recalling topK recalled text fragments most related to user input; and for each recalled text fragment, according to the corresponding document directory structure, obtaining a vectorized text fragment corresponding to the same level or higher level of the recalled text fragment, and obtaining a recalled long text meeting the length range of the recalled text.

The invention belongs to the field of image text segmentation in the field of computer vision, and discloses an image text segmentation method based on frequency separation. Firstly, a CNN and a Transform are used to construct a local feature coding branch and a global feature coding branch respectively, and local features and global features from an image are extracted preliminarily. In the frequency feature enhancement module, wavelet transform is used to decompose the frequency of the input feature. And then enhancing the obtained high-frequency component and low-frequency component by using different strategies. And finally, low-frequency information is utilized to guide the network to pay attention to local features near the text region, the influence of irrelevant background noise on a segmentation result is reduced, and finally an accurate segmentation result is obtained. According to the method, the problem of dependence on a text detection model in an image text segmentation task is solved, and a result output by the method has high integrity and accuracy.