274 results about "Lexical frequency" patented technology

The invention provides an unstructured text data calculation method and system, and relates to the technical field of natural language processing, and the method comprises the steps: carrying out the multi-level processing of input unstructured text data, dynamically adjusting the word segmentation granularity according to the word frequency distribution, constructing a part-of-speech co-occurrence matrix in combination with context semantic information, and extracting entity information. And fusing the part-of-speech co-occurrence matrix and the entity information to generate a hierarchical semantic tag sequence. Feature extraction units with different convolution kernel sizes are adopted to extract feature representation, and cosine similarity between different semantic hierarchies is calculated to establish an association weight matrix. A semantic enhancement vector is constructed based on entity information, and adversarial training is carried out to obtain a multi-modal semantic feature matrix. And calculating the semantic similarity between the fusion feature vectors for clustering, setting difficulty weight sorting according to the complexity, consistency and ambiguity of class clusters, then inputting the class clusters into a classifier, and iteratively optimizing a classification result to obtain a calculation result.

The invention provides a semantic input-based artistic font generation method, which comprises the following steps of: acquiring text data required by artistic font generation, and performing word segmentation processing on a text by adopting a word segmentation method to obtain a keyword list; performing semantic extension on each keyword in the keyword list according to a pre-constructed HowNet knowledge base to obtain synonyms, hypernyms and hyponyms of the keywords; performing word frequency statistics and TF-IDF weight calculation on the expanded keyword list, and screening out a core semantic concept required by the keyword art font generation with the highest weight; calculating semantic similarity among the core semantic concepts, and constructing a semantic similarity matrix; taking the screened core semantic concepts as nodes and the semantic similarity as weights of edges, and constructing a semantic network atlas; k-means clustering analysis is carried out on the semantic concept vectors, concepts with high semantic similarity are clustered according to a clustering result, and a semantic vector representation model required by artistic font generation is constructed.

The invention discloses a cue word compression method and system based on TF-IDF and cross entropy, belongs to the technical field of large model cue word compression, and aims to solve the problems that redundant information is introduced into long cue words, the model efficiency is reduced and the cost is increased. To-be-compressed content is divided into sentences at the sentence level and then converted into embedded vectors, and the Euclidean distance is calculated in combination with problem vectors so as to screen related sentences; calculating a TF-IDF value at the word level through a word frequency and an inverse document frequency to extract keywords and recombine sentences; and selecting a reference model and a basic model at the Token level, identifying the key Token based on a cross entropy loss difference value, and splicing the key Token in sequence to generate a compressed cue word. According to the method, a complex calculation structure is avoided, the inference efficiency is improved while the semantic integrity is maintained, and the resource consumption is reduced.

The invention belongs to the field of artificial intelligence, and particularly relates to a watermark generation method and system based on a large language model, and the method comprises the steps: obtaining a training data set, and carrying out the preprocessing of the data in the training data set; presetting a watermark coding rule; training a large oracle model by adopting the preprocessed data according to a watermark coding rule; obtaining to-be-processed text data; cleaning and standardizing the text data; calculating a word frequency and an inverse document frequency of the standardized text data, and generating an initial vector; processing the initial vector by adopting a Word2Vec model to obtain a high-dimensional word vector; inputting the high-dimensional word vector into the trained neural network model, and extracting deep features; classifying the texts according to the deep features; inputting the classified questions into the trained large oracle model, and generating watermarks corresponding to the text types; according to the method, a watermark coding rule is designed, and the corresponding watermark is generated for the classified text, so that the safety of the text is improved.

The invention provides a medical record information extraction and analysis method and system based on a deep learning large model, and the method comprises the steps: carrying out the semantic understanding of a medical record text through a deep learning model, and recognizing key clinical features; extracting the key clinical features, and carrying out vectorization processing on the extracted key clinical features; performing sentence segmentation and word segmentation on the case text based on a rule engine and the key clinical features; adjusting the segmented words based on a symptom classification rule; and carrying out word frequency statistics on the adjusted segmented words. According to the method, accurate recognition and structured conversion of key clinical features can be ensured through a semantic understanding technology, knowledge graph construction and machine learning model optimization are supported, and a new view angle and a new tool are provided for exploration and research of diseases.

The invention belongs to the field of vehicle industry design, and discloses an electric bicycle form design method and system, electronic equipment and a storage medium, and the method comprises the steps: mining user emotion vocabularies through online comments, constructing an emotion lexicon in combination with an improved word frequency-inverse document frequency algorithm and a D-S evidence theory, and screening out key perceptual vocabularies; constructing a convolutional long-short-term memory neural network model optimized by a snake swarm algorithm to realize a mapping model between customer sensibility and product morphological characteristics; carrying out subjective and objective comprehensive evaluation on the design scheme in combination with an eye movement experiment and subjective evaluation so as to screen out an optimal design scheme; an optimal scheme is selected and input into a generative AI platform for multi-angle visual rendering, ergonomic modeling and aerodynamics simulation are assisted, and the structural feasibility and performance are verified. The method provides systematic technical support for emotional value improvement and design optimization of industrial products.

The invention discloses a large model-based government affair knowledge retrieval method, apparatus and device, and a medium, and relates to the technical field of natural language processing, and the method comprises the steps of performing information extraction on a to-be-processed government affair work order through a preset government affair large model to obtain target structured information; based on a word processing fusion algorithm constructed by a word frequency-inverse document frequency algorithm and a graph sorting algorithm, performing word processing on the target structured information to obtain work order keywords; converting the work order keyword into a target semantic vector; performing retrieval operation on a preset government affair knowledge base by utilizing a preset retrieval condition and the target semantic vector to obtain a target government affair knowledge document; the preset retrieval condition is a retrieval condition constructed based on time, space and the work order emergency degree. Therefore, the information accuracy and the processing efficiency can be improved through word processing; and in combination with retrieval conditions constructed based on time, space and work order emergency degrees, accurate retrieval can be provided, and the processing effect on various complex government affair work orders is improved.

The invention discloses a big language model training method based on a knowledge graph, which comprises the following steps: collecting professional text data in a specific field, extracting entities and relationships from the professional text data, constructing a field knowledge graph, and constructing the knowledge graph based on data of different data sources on the basis of an external network; and introducing the domain knowledge graph into an intranet, storing the domain knowledge graph into a knowledge base in the intranet, extracting terminologies from the domain knowledge graph, analyzing word frequencies of the terminologies in the professional text data, and selecting the terminologies with the word frequencies higher than a set threshold value as newly added high-frequency terminologies. According to the method, the professional knowledge graph of the target field is constructed, the big language model is finely adjusted based on the professional knowledge graph, and the professional knowledge big language model is generated.

The invention discloses a lightweight knowledge graph rapid construction method and system based on an NLP technology. The method comprises the steps of preprocessing an unstructured text and segmenting the unstructured text into semantic segments; unsupervised clustering is combined with the contour coefficient to determine the optimal clustering number, and a semantic association fragment cluster is obtained; performing word segmentation, part-of-speech tagging, NER and entity linking on the fragment cluster, extracting an entity and initial relationship, and fusing semantic similarity, TF-IDF word frequency collaboration degree and co-occurrence frequency to calculate a relationship edge weight; constructing a lightweight knowledge graph; in the question and answer stage, questions are disassembled through a process engine, related sub-graphs are retrieved, and answers with reasoning links are generated. The system correspondingly comprises a text preprocessing module, a clustering module, an entity relation processing module, a graph construction module, a question and answer reasoning module and a storage module. According to the method, the construction cost is reduced, the interpretability and the module coupling degree are improved, multiple scenes such as government and enterprise public opinions and medical assistance are adapted, and the problems of weak generalization, poor real-time performance and'black box 'in the traditional technology are solved.

The invention relates to the technical field of aggregation retrieval, in particular to a distributed aggregation retrieval method for a law and regulation text database, which comprises the following steps: collecting and integrating law and regulation text data, preprocessing the law and regulation text data, removing useless punctuations and blank characters, and generating a preprocessed text set. According to the method, data integration and preprocessing are carried out on the regulation text, useless punctuations and redundant blank characters are effectively removed, the purity of text data is guaranteed, and the accuracy of subsequent word segmentation and word frequency statistics is improved; the implementation of the word segmentation and word frequency statistical process is helpful for identifying key feature vocabularies in the regulatory text, so that the accuracy of the subsequent inverted index construction stage is enhanced; by preliminarily constructing the inverted index and implementing similar item merging and low-frequency vocabulary removing operation, the index scale and redundant interference are reduced, and the index query efficiency and accuracy are improved.

The invention discloses a retrieval enhancement generation optimization method and device based on adaptive classification and autonomous reasoning. The method comprises the following steps: constructing a domain knowledge vector database; inputting a proprietary domain problem, and finding a plurality of matched knowledge documents with the highest score in the vector database from two dimensions of semantics and word frequency; inputting the matched knowledge document into an adaptive classification module to obtain a semantic attribute category of the document; the document and the corresponding semantic attribute category are input into an autonomous reasoning module, uncertain items and error items are eliminated, and enhanced knowledge is obtained; the enhanced knowledge is sent to the intelligent question and answer module to generate a response answer; the response answers are sent to the self-adaptive classification module again, and semantic attribute categories of the response answers are checked; if the answer is not judged to be correct, the autonomous reasoning module and the subsequent steps are executed again on the response answer; and if the answer is judged to be correct, displaying the answer to the user. The problem that in the prior art, an external knowledge source has negative effects on a generation result can be solved.

The present invention discloses a method and device for evaluating a multi-modal large model for urban governance based on multi-level texts, comprising the following steps: obtaining the true label text of a sample picture and the output text of the multi-modal large model for urban governance; using an urban management thesaurus to segment the text into independent words and calculating the similarity score of lexical units; using the urban management thesaurus to create a bag-of-words model to convert the text data into a word frequency matrix and calculating the low-level word frequency similarity score; using a trained language model to extract semantic features with context in the text and converting them into vector representations, and calculating the semantic similarity score; using a general large language model with a formatted request with business guiding words and evaluation requirements to obtain the high-level semantic similarity score corresponding to the text; and combining the similarity scores at each level to calculate the comprehensive evaluation score of the multi-modal large model. The present invention fully considers the similarity measurement in the multi-level space of texts, improving the accuracy of evaluating the text output of the urban governance large model.

The invention relates to the field of intelligent report generation, in particular to an intelligent insight report generation method and system based on an AI large model, and the method comprises the steps: inputting an insight demand, and generating an insight data package comprising insight contents, associated data and industry labels; extracting a basic keyword set of the insight data packet to form a mixed feature code; after mixed feature coding preprocessing, weight distribution is carried out; and after weight distribution of the AI large model, injecting a high-weight feature vector into a semantic understanding core layer, injecting a low-weight feature vector into a logical reasoning layer, and outputting analysis data to form an intelligent insight report. According to the method, word embedding parameters are optimized according to field semantic characteristics, a parameter verification mechanism is introduced, field text characteristics are adapted by adjusting vector dimensions, context windows and low-frequency vocabulary filtering threshold values, window parameter validity is verified through cosine similarity, word frequency threshold value reasonability is verified through standard deviation, and field text characteristic matching is achieved. And it is ensured that the feature vectors can accurately capture domain-specific semantics.

The invention discloses a method and system for realizing data classification and grading based on a large language model technology, and the method comprises the steps: collecting related data from a plurality of data sources, and carrying out the preprocessing, thereby forming a standardized preprocessing data set; obtaining text data from the text data, and extracting features of word frequency, TF-IDF values and semantic vectors to obtain a comprehensive feature vector; randomly extracting a part of the standardized preprocessed data set, and dividing and labeling the standardized preprocessed data set to obtain labeled training set data and test set data; training a classification and grading model; an optimal classification and grading model is selected; inputting the data to be classified and graded into the optimal classification and grading model to obtain a classification and grading result; the system comprises a data acquisition module, a data processing module, a feature extraction module, a data labeling module, a model training module, a model evaluation module and a data classification and grading module. According to the method, the scientificity, accuracy and efficiency of classification and grading are improved, and the requirements of enterprises and organizations for data classification and grading management are met.

An archive retrieval method, system and product and a readable storage medium relate to the field of archive retrieval, and the method comprises the following steps: performing initial matching based on a water conservancy project professional retrieval word, obtaining a professional term dictionary and a business rule template from a water conservancy project corpus, performing word frequency statistics and keyword extraction by using the professional term dictionary, and obtaining a business rule template; and matching the extracted business keywords with the standard terms to obtain professional feature items, substituting the professional feature items into the business rule template for rule matching, determining the engineering stage, the technical index and the construction process of the engineering technology document, and generating an archive business label. And according to the professional role type of the user, selecting a corresponding service label combination for recombination and display. By implementing the method, the correlation between the retrieval result and the user can be improved.

The invention discloses a retrieval optimization method based on dynamic mixed retrieval and adjacent paragraph introduction. The method comprises the following steps: taking related documents as a corpus of a retriever; performing retrieval by using a BM25 algorithm and a HyDE algorithm to generate corresponding preliminary retrieval document paragraphs; calculating a query specificity value for user query by using TF-IDF; determining weights of a BM25 algorithm and a HyDE algorithm, and respectively performing weighted fusion on a document paragraph correlation score based on word frequency statistical calculation in the BM25 algorithm and a document paragraph correlation score based on semantic embedding similarity calculation in the HyDE algorithm to generate candidate document paragraphs and comprehensive scores corresponding to the candidate document paragraphs; the document paragraphs ranked in the top are selected from the preliminary retrieval document paragraphs, adjacent document paragraphs are obtained, weighted scores of the adjacent document paragraphs are calculated, the adjacent document paragraphs are added into a candidate document paragraph list, and the document paragraphs with the top scores are selected from the list to serve as retrieval results.

The invention discloses a science and technology theme evolution stage prediction method and system based on multi-graph representation. The method comprises the following steps: firstly, carrying out subject matching on science and technology subject keywords in science and technology literatures, screening out the science and technology subject keywords capable of representing a subject, then obtaining a word frequency sequence of the science and technology subject keywords, and extracting semantic association relationship characteristics among the science and technology subject keywords and reference relationship characteristics among the science and technology subject keywords; secondly, a science and technology theme keyword multi-graph network is constructed, and science and technology theme keywords representing themes are modeled; then, for features of different dimensions, different graph neural network models are adopted for feature extraction and modeling; and finally, predicting the evolution stage of the science and technology subject by using the prediction model, thereby solving the technical problem that the analysis granularity is too coarse in the research of the evolution of the science and technology subject in the prior art, and realizing the accurate prediction of the evolution process of the science and technology subject.

The invention discloses a resume label generation method and device and a medium, and relates to the field of natural language processing, and the method comprises the steps: carrying out the preprocessing of an original resume text, and carrying out the word segmentation of original resume data into a plurality of text words; counting word frequencies corresponding to the text words, and endowing the text words with corresponding importance weights according to the word frequencies; vectorizing the text words to obtain corresponding word vectors, and aggregating the word vectors according to the importance weight to obtain resume text vectors corresponding to the original resume text; based on a preset hierarchical label library, performing multi-label classification on the text semantic vector through a nonlinear classification algorithm, and outputting a prediction label of the original resume text; and screening the label prediction values based on a preset threshold to obtain a final resume label set. By fusing word frequency weighted semantic representation and a nonlinear classification algorithm, the accuracy and adaptability of resume label generation are remarkably improved.

The invention provides a document classification method, which belongs to the field of data classification, and comprises the following steps: collecting original documents of different languages from a plurality of channels, and converting texts of different languages into a unified language representation form; extracting keywords in the document after language conversion, calculating word frequency and inverse document frequency of each keyword in the corresponding document, weighting the word frequency by adopting the inverse document frequency to obtain a plurality of TF-IDF weights, and converting the plurality of TF-IDF weights into a plurality of word embedding features; meanwhile, text data similar to the embedding features of the multiple words in the original document are labeled; inputting the plurality of word embedding features into a natural language processing model to train the model, and training the trained model again through the labeled document data to obtain a semantic encoder; and inputting the to-be-classified documents of different languages into a semantic encoder, and outputting a classification result. According to the method, the documents of different languages can be accurately classified by adopting two hierarchical training models, so that the document classification efficiency is improved.

The invention discloses an AI accompanying system and method based on general artificial intelligence, and the method comprises the following steps: S1, collecting and preprocessing user data, and generating a standardized data set; s2, performing dimension reduction by adopting diffusion mapping, extracting voice and behavior characteristics, and retaining a state change trend; s3, constructing a behavior knowledge graph based on symbol logical reasoning, and recording symbol state changes; s4, similar states and coping strategies are retrieved by using sparse distributed memory storage symbol representation; s5, combining manifold embedding, historical matching and symbolic reasoning, updating a user state and generating a personalized accompanying strategy; s6, analyzing the voice rhythm and the word frequency, constructing an emotion evaluation model, and generating a personalized emotion intervention scheme; s7, optimizing the knowledge graph by adopting incremental memory, and improving the companion strategy adaptability; and S8, recording feedback, continuously optimizing emotion intervention, and realizing intelligent accompanying. The AI accompanying system improves intelligence, personalization and long-term adaptability of AI accompanying, and is suitable for the fields of old-age care, mental health management and the like.

The invention relates to the technical field of natural language processing, in particular to a semantic understanding-based completion environmental protection acceptance survey report auxiliary auditing system, which comprises a structure recognition module, a semantic chain construction module, a semantic reconstruction module, an image-text coordinate fusion module and a spatial offset judgment module. In the method, through extracting subjects, terms and facility words in facility paragraphs and establishing a combination relationship, ordered labeling of semantic elements is realized, statement structure definition is enhanced, a semantic chain group is constructed and facility classification is associated, semantic consistency comparison is supported, semantic sequences and word frequency distribution in design and reply contents are compared, and description consistency is verified; facility coordinates in an image are extracted, character paragraphs are matched, image-text content anchoring, orientation, terrain and distance word extraction and conversion into space parameters to be compared with an image path are achieved, space consistency verification is completed, and the intelligent level of auditing on semantic comprehension, image-text fusion and space logic is overall improved.

The invention provides a hot event mining method and system, and relates to the technical field of natural language processing, and the method comprises the steps: obtaining news website data comprising a plurality of pieces of news, and vectorizing a news title of each piece of news to obtain a title vector; based on the title vector, dividing all news of the news website data into a plurality of clusters through a clustering method; performing importance evaluation on news in each cluster according to the news content, then selecting a most important news title from each cluster, and extracting a news trunk from the most important news title to form a candidate event list; the number of news in the cluster is used as the popularity of the events, and the events in the candidate event list are sorted, so that final hot events are screened out; according to the method, the candidate events are selected based on the word frequency-reverse file frequency of the keyword, and information in a news title is concisely focused by extracting a news trunk, so that the accuracy and robustness of hot event mining are improved.

The invention relates to the technical field of text processing, in particular to an internet-oriented illegal advertisement recognition method, device and system. According to the method, text information data carried by an advertisement is specifically analyzed, the high distribution consistency condition of part-of-speech and word frequency of each segmented word in a local text before and after special symbol processing and the similarity condition of an overall text are compared and analyzed, and the necessary processing degree of a special symbol is obtained in combination with the connection association condition of words before and after the special symbol processing; through the processed text data and the abnormal possibility of the image and the release time, the violation score of the advertisement is evaluated and identified, and a more accurate evaluation result is obtained. According to the method, by comparing semantic change conditions represented by coherent text contents before and after special symbol processing, interference special symbols carried in the advertisement text are processed, the accuracy of advertisement violation information identification is improved, and violation advertisement identification is more comprehensive and reliable.

The invention provides a text data parallel semantic deduplication method, system and device and a medium, and belongs to the technical field of text processing. The method comprises the steps of collecting large-scale text data to be subjected to duplicate removal, performing preprocessing to generate sample data, performing word segmentation and word frequency statistics by using a BERT word list, and calculating a word frequency vector of each piece of sample data; using a pre-trained BERT model to extract semantic features of the sample data, calculating word semantic weights of words in the sample data according to the word frequency vectors, and calculating similarity of the sample data based on the semantic features and a simhash algorithm; and according to the semantic similarity of the sample data, maintaining a pre-deletion dictionary and a global deduplication graph, and executing a deletion operation of the sample data based on the global deduplication graph to form a deduplicated data set. By combining the BERT model, the simhash algorithm and the global deduplication graph, the efficiency and accuracy of parallel semantic deduplication of large-scale text data are effectively improved.

The invention provides a professional lexicon construction method for coal industry NLP word segmentation, and relates to the technical field of lexicon construction, and the method comprises the steps: carrying out coal industry keyword extraction on coal periodical data, carrying out first word frequency sorting on a plurality of extracted coal industry segmented words, and carrying out second word frequency sorting on a plurality of extracted coal industry segmented words; constructing a first word bank and a second word bank according to the first word frequency sorting; coal industry standard terms are recognized through a character recognition technology to construct a third word library; based on a deep learning model, expanding the second lexicon and the third lexicon to obtain an initial lexicon; performing second word frequency sorting in the initial word bank, and constructing a fourth word bank according to the second word frequency sorting; and performing repeated word segmentation screening on a fifth word bank formed by the fourth word bank and the first word bank to obtain the coal industry word segmentation word bank. Therefore, the quality of the NLP word segmentation result in the coal industry is improved, and the accuracy of processing large-scale data by a large model in the coal industry is enhanced.

The invention discloses an extended query method and system based on big language model entity extraction, key entities in original queries are extracted, the original queries are repeatedly spliced, the spliced original queries and key entities are fused, extended queries are obtained, entity interpretation is used as an extension basis, and compared with a traditional word frequency statistics or synonym replacement method, the extended query method and system have the advantages that the extension efficiency is improved; the large language model can generate extended fragments with semantic consistency and information integrity in combination with the context, it is guaranteed that the finally generated extended query is consistent with the original intention of a user, semantic drift and invalid extension are reduced, the correlation and precision of extended content are improved, meanwhile, the spliced original query and key entities are fused, and the user experience is improved. According to the method, the original query is locally enhanced by adopting an entity-level semantic interpretation mode, higher interpretability and controllability are prepared, and a complicated model structure and a training process are not needed, so that the deployability and universality of the method are remarkably improved, the expansion effect is more stable, and diversified retrieval requirements can be met.

A method initializes and accelerates training of neural network based large language model, including by: (i) accessing a corpora for training a neural-network based large language model having word embeddings and word projections in respective word embedding and word projection layers and at least one hidden layer; (ii) counting raw token frequencies associated with content within the corpora; (iii) smoothing the raw token frequencies into a series of vector norms based on log or scaled log functions parameterized by maximum norm information; and (iv) injecting vector norm information into word embeddings and / or word projections based on norm-angle reparameterization to prepare the large language model for training.

The invention discloses a data mining system and method for mail data, relates to the technical field of data retrieval processing, solves the technical problem of inaccurate retrieval matching caused by single semantic analysis retrieval, and improves data relevance by unifying multi-source data formats and combining an enterprise address book to supplement metadata. Structured data standardization and unstructured text vectorization are carried out, a computable semantic space is constructed, keywords are screened through a matching proportion, a word frequency weight is calculated in combination with a BM25 algorithm, the keyword matching accuracy is improved, a semantic vector is generated by utilizing a pre-training language model, and semantic correlation is calculated through cosine similarity. Keyword matching scores and semantic similarity scores are dynamically fused, accuracy and comprehensiveness are balanced, accuracy and recall rate are calculated in real time, retrieval deviation types are automatically recognized, keyword weights and similarity threshold values are adjusted in a targeted mode or synonym lists are expanded, and self-adaptive optimization of the system is achieved.