277 results about "Tf–idf" patented technology

The invention relates to a realization method and system for electronic medical record post-structuring and auxiliary diagnosis. A combination mode of multiple types of distance measurement is used: a character string editing distance refers to a minimum number of replacement, insertion and deletion operations required for converting a character into another character string; a Jaro-Winkler distance measures similarity between two character strings and is used for repeated recording detection; a geometric mean value of a Chinese character distance and a Chinese character input method is adopted as comprehensive similarity measurement for measuring similarity between characteristic texts; characteristic ranking is realized by using a TF-IDF method and is used for assessing the importance of characteristic terms relative to documents in a file set or a corpus library, and the importance of the characteristic terms is in direct proportion to an occurrence frequency in the documents and is in inverse proportion to an occurrence document in the corpus library; and files are converted to be in a file format of PU learning of a positive example data set and an unlabelled data set according to the generated characteristic terms, and through the PU learning, the system automatically recommends related diagnoses for clinical medical personnel to refer.

The invention discloses a natural language processing-based multi-language analysis method and device. The method comprises the following steps of: selecting to input a natural language text information language category through a language detection training model; obtaining word embedding expression information of corresponding words which can be recognized by a computer through a trained word vector model, and extracting a keyword of the obtained word embedding expression information through a TF-IDF manner; calculating an article vector and a category vector of each preset category according to the keyword and a keyword weight, and calculating a similarity between an article of natural language text information and each preset category so as to determine a text classification result ofthe natural language text information; and inputting the word embedding expression information of the natural language text information into a trained convolutional neural network and a parallel-framework text emotion analysis model of a bidirectional gate circulation unit, and obtaining a final emotion tendency value through calculation. According to the method and device, the problem that traditional multi-language analysis method needs to know domain knowledges of related linguistics and needs plenty of manpower to carry out operation is solved.

The invention provides a telecommunication field package recommending method based on intelligent customer service robot interaction. The method comprises the following steps: a. acquiring a user interest model; and b. recommending individual demand-satisfied package service for a user by adopting a decision tree algorithm. The invention further provides an intelligent customer service robot system recommending engine device. Compared with the existing recommending method, the telecommunication field package recommending method is based on a scene interaction model in a recommending process, and can be used for carrying out similarity calculation according to similarity among the calculation labels, calculating the similarity among the labels by combining similarity between semantic similarity calculation with the traditional TF-IDF (Term Frequency-Inverse Document Frequency) in the similarity calculation, better reflecting the characteristics of users and resources by applying labels to show resource and user models, and improving the recommending quality.

The invention discloses a network hot event detection method based on text classification and clustering analysis. The method solves the problem that the efficiency and accuracy rate of the existing network hot event detection method based on clustering analysis need to be improved. The method comprises the steps that feature words are respectively selected for various classes of files through feature extraction and feature selection by utilizing a training corpus; each training text and test text are represented as vectors in all of the feature spaces by utilizing a vector space model method, and the weight of each dimension of the vectors is determined by utilizing a TF-IDF (term frequency-inverse document frequency) method, and then each test text is classified; the classified test texts in different classes are respectively subjected to clustering analysis, so the hot cluster of each class is obtained, the feature word representing the hot event is obtained through further analysis, and then the word property and other aspects of each feature word are analyzed; the description of each hot event is generated by utilizing relevant language knowledge and necessary linguistic organization. With the network hot event detection method based on text classification and clustering analysis, the detection efficiency and accuracy rate of hot events can be effectively improved.

The invention provides a text information similarity matching method and device on the basis of TF-IDF. The method includes the steps of acquiring text information; performing word segmentation on thetext information to obtain word segments w1, w2, ..., wn-1 and wn; using a CBOW model for calculating word vectors V(w1), V(w2), ..., V(wn-1) and V(wn) of the word segments respectively; using the TF-IDF algorithm for calculating TF-IDF values k1, k2, ..., kn-1 and kn of the word segments respectively; according to products of the word vectors of the word segments and the corresponding TF-IDF values, obtaining a sentence vector V; calculating the cosine similarity between the sentence vector V and sentence vectors of pre-stored statements, and determining the pre-stored statement with the maximum cosine similarity. Through the process above, the pre-stored statement which is the most similar to the text information can be found, the accuracy of problem recognition can be improved in the aspects of robot dialogue, information classification and the like, and therefore the dialogue efficiency or the classification efficiency is improved. A computer device and a storage medium are also provided.

The invention discloses an automatic microblog text abstracting method based on unsupervised key binary word extraction. The automatic microblog text abstracting method comprises the steps of preprocessing a microblog; standardizing a binary word; extracting a key binary word based on a mixed TF-IDF (term frequency-inverse document frequency), TexRank and an LDA (local data area); sequencing sentences based on the intersection similarity and a mutual information strategy; extracting abstract sentences based on a similarity threshold value; generating abstract by reasonably combining the abstract sentences. According to the automatic microblog text abstracting method, the binary word is used as a minimum vocabulary unit, and the binary word has richer text information than words, so that the sentences based on the key binary word is higher in noise immunity and accuracy than the sentences based on key word extraction; meanwhile, when the abstract sentences are extracted, the similarity threshold value is introduced to control redundancy, so that the abstract is higher in recall rate. The abstract generated by the method is accurate, simple and comprehensive; the efficiency and the quality that a user acquires knowledge are obviously improved, and the time of the user is greatly saved.

The invention discloses a junk mail filtering method based on support vector machine (SVM). The steps are as following: 1) analyze the mail and extract the message relevant to title, text and character set; 2) carry out divided syncopation to the extracted text message content; 3) make statistics of word frequency in mail and utilize TF-IDF formula to map the mail text to vector; 4) utilize LibSVM to train the mail sample and obtain support vector machine model; 5) utilize support vector machine model to classify new mail and obtain the probability value of junk mails; 6) utilize threshold value adjustment to guarantee a lower level of false positive rate of normal mails to junk mails and ultimately judge whether mails are junk mails. The invention utilizes the advantage of highest single model classification accuracy of the support vector machine, improves the correctness of junk mail filtering, according to the text feature and activity feature and at the same time, also effectively solves the problem of unequal miscarriage cost in junk mail filtering.

The invention discloses a filtering method of recursive Chinese junk E-mail, which is based on Logistic. The method comprises the following steps: first, analyzing E-mails, extracting E-mail titles, E-mail main bodies and accessory relative information, second, segmenting words for version information which is extracted, third, accounting word frequencies of entries in E-mails, calculating weights of words through utilizing TF-IDF pattern, presenting the E-mail to be characteristic vector which is weighted, fourth, utilizing an LIBLINEAR tool kit to exercise the sample of the E-mail to get an Logistic recursive module, fifth, utilizing the Logistic recursive module to classify for new E-mails, getting the probability value whether the E-mails which are got are junk E-mails. The utility which utilizes the Logistic recursive module has the advantages of simple module, little amount of parameter, and high classifying accuracy in a data set whose text number and characteristic number are both bigger, the accuracy and efficiency of filtering junk E-mails are improved through dimension reduction and improved characteristic value calculating method, and meanwhile, the problem of choosing module exercise parameter which is faced in filtering junk E-mails is effectively solved.

The invention discloses a topic feature text keyword extraction method. Through the method, text keyword extraction results better than those of a traditional TF-IDF method can be obtained. Accordingto the technical scheme, at a training stage, word segmentation, stop word removal, part-of-speech filtering and other preprocessing are performed on a training text, statistical analysis is performedon inverse document frequency of words, meanwhile a topic model method is utilized to learn and obtain a topic probability matrix of the words, normalization processing is performed, topic distribution entropy of the words is calculated according to the topic probability matrix of the words, global weights of the words are calculated in combination with the inverse document frequency and the topic distribution entropy, and global weight calculation results are output to a test stage; and after a test text is preprocessed, statistical analysis is performed on normalized term frequency of wordsin the test text, the normalized term frequency is combined with the global weight calculation results obtained at the training stage, comprehensive scores of the words are calculated are ordered, and a plurality of words with the highest scores in the score order are used as automatic keyword extraction results of the current test text.

The invention discloses a method and a system for recommending text files. The method includes: determining a term set of a current text file and then determining a TF (term frequency) value or a TF-IDF (inverse document frequency) value of each term in the term set; determining an implied subject feature vector of the current text file; respectively computing similarity degrees among the implied subject feature vector of the current text document and implied subject feature vectors of various text files to be recommended; and selecting certain text files to be recommended and recommending the certain text files to be recommended. The similarity degrees among the implied subject feature vectors of the certain text files to be recommended and the implied subject feature vector of the current text file meet preset screening conditions. The method and the system have the advantage that the similarity degrees among the text files are computed from the implied subject feature vectors, so that the method implemented by the aid of the system for recommending the text files is more accurate.

A text semantic analysis method and system can realize semantic analysis of text data base on lexical level and sentence level. Aiming at the semantic analysis at the lexical level, the invention firstly adopts an improved word segmentation algorithm to solve the problem that English words are segmented only by spaces. Secondly, based on word segmentation, TF-IDF modeling is performed to obtain weight value; Then the text is vectorized by weighting and summing the weight value and the word vector trained by Word2Vec, and finally the document similarity is solved. At the same time, the invention considers the contribution degree of the vocabulary to the document content and the semantic status to calculate the similarity degree of the document, the result has higher accuracy, and provide agood foundation for subsequent text clustering. The present invention extracts subject-predicate object structure based on text segmentation, part-of-speech tagging, syntactic analysis and dependencyrelation for sentence level semantic analysis. The invention realizes the extraction of subject-predicate-object structures of various sentence types in all aspects, and realizes the noun expansion function, which is more consistent with the manual extraction result.

The invention discloses a method and a device for generating text characteristic vectors based on TF-IGM, as well as a method and a device for classifying texts. The concentration ratios of characteristic words distributed in different classes of texts are calculated by establishing inverted gravitational moment (IGM) models, and the weights of the characteristic words are calculated based thereon. The weights obtained by the calculation can more realistically reflect the importance of the characteristic words in the text classes, accordingly increasing the performance of text classifiers. The device for generating the text characteristic vectors based on the TF-IGM has a plurality of options that may be optimized and regulated based on the results of the performance test of the text classes in order to be adaptive to text data sets having different characteristics. It is proved by experiments on public English corpus and Chinese corpus that the TF-IGM method is much more superior to the existing methods such as TF-IDF methods and TF-RF methods, and the TF-IGM method is particularly applicable to multi-class text classifications of more than two classes.

The invention discloses a feature bag image retrieval method based on a Hash binary code. The method comprises steps that, a vision term list is established; tf-idf (term frequency-inverse document frequency index) weight quantification of vision terms is carried out; vision term characteristic quantification of an image is carried out; an inverted index is established; a feature binary code projection direction is learned; feature binary code quantification is carried out; candidate image sets are retrieved. According to the method, the index is established for an image database, rapid image retrieval is realized, and retrieval efficiency is improved, moreover, through a binary code learning method having the similarity retention capability, the binary code is learned from spatial distance similarity and meaning distance similarity as signature, and image retrieval accuracy is improved. The feature bag image retrieval technology based on the Hash binary code has properties of high efficiency and accuracy, and relatively high use values are realized.

The invention discloses a TF-IDF feature extraction based short text classification method. According to the method, short texts are merged into a long text so as to enhance the TF-IDF feature of the short texts; dimension reduction is performed so as to generate a feature word list and a feature word dictionary; a mechanism compensation is established for a class having a relative unobvious feature while the feature word list is established, and the text feature vector weight is enhanced; and other word banks or word vector dictionaries do not have to be constructed or trained, and then the algorithm performance can be improved on the premise of ensuring the feature expression result of the texts. The TF-IDF feature extraction based short text classification method can be widely applied to the field of data processing.

The invention discloses a problem similarity calculation method based on a plurality of features, includes steps: For the input new question sentence, Compared with the stored historical questions andcorresponding answers, the similarities between the new questions and the historical questions are calculated based on character features, semantic features of words, semantic features of sentences,implied topic features of sentences and semantic features of answers. The final similarity is the product of the above five similarities and their corresponding weights, which are trained by linear regression method. The invention adopts a plurality of features to increase the diversity of sample attributes, and improves the generalization ability of the model. At that same time, the soft cosine distance is utilized to convert the TF-IDF is fused with editing distance, word semantics and other information, which overcomes the semantic gap between words and improves the accuracy of similarity calculation.

The present invention discloses a microblog data analysis based hot news prediction method and system. The method comprises: acquiring news reports from mainstream news sites and microblog user response information caused by the news reports on the microblog; carrying out word segmentation and word frequency statistics to a microblog text, calculating a TF-IDF value of a word, and converting the value into a microblog topic described by using a vector space; classifying the microblog topics, counting each quantitative index for describing the microblog topics, and calculating each hot index of news; and using a multivariate linear regression algorithm to learn sample data, establishing a hot news prediction model, and determining whether the latter news can become a hot news or not. The system comprises a data acquisition module, a text analysis processing module, a data statistical analysis module and a hot news prediction module. According to the method and system disclosed by the present invention, the trend of news reported by media in microblog topics is comprehensively analyzed to predict whether the news can become a hot news or not in public sentiments, so that the problem of early prediction of hot news can be well solved.

Content Based Similarity Detection. A computer implemented method includes computing a hash of each word in a collection of books to produce a numerical integer token using a reduced representation and computing an Inverse Document Frequency (IDF) vector comprising the number of books the token appears in, for every token in the collection of books. The method also includes creating a token occurrence count vector for each book in the collection and normalizing the token occurrence count vector using the IDF vector to create a Term Frequency-Inverse Document Frequency (TF-IDF) vector. Further, the method includes reducing each TF-IDF vector by using random projections to obtain a final signature representing each book in the collection, reducing each TF-IDF vector by using random projections to obtain a final signature representing each book in the collection and using a trained machine learning algorithm, determining whether each of the list of candidate books is similar to the target book.

The invention discloses a hot word finding method, and particularly relates to a public opinion hot word finding method based on a keyword weighting algorithm. According to the public opinion hot word finding method based on the keyword weighting algorithm, a Chinese word segmentation tool is utilized to conduct preliminary word segmentation on massive public opinion information, part-of-speech tagging is provided, an IDF table, a filter word table and a part-of-speech weighting value table are combined at the same time, according to a weighting type TF-IDF algorithm, a candidate word popularity value is calculated, the calculation is not only relied on word frequency, instead, effective information contained in part-of-speech, position and the like of a word is taken into full account, and reliability basis is provided for hot word recognition. In addition, in the public opinion hot word finding method based on the keyword weighting algorithm, the characteristic that the public opinion has a distinct topic and theme under a we media time is taken into full account, corpus processing is mainly conducted on the public opinion topic, and the problem of the efficiency of the hot word recognition under massive public opinion information is solved. Finally, dynamic incremental type updating is achieved for the IDF table, the real-time performance of the word inverse document frequency is guaranteed, and the accuracy of the hot word recognition is improved.

The invention discloses a text clustering multi-document automatic abstracting method and a system for improving a word vector model. The CBOW of the Hierachic Softmax belongs to the field of large-scale model training, and the CBOW of the Hierachic Softmax belongs to the field of large-scale model training. Based on the method, a TesorFlow deep learning framework is introduced into word vector model training; the problem of time efficiency of a large-scale training set is solved through streaming processing calculation, TF-IDF is introduced firstly during sentence vector representation, thenthe semantic similarity of a semantic unit to be extracted is calculated, weighting parameters are set for comprehensive consideration, and a semantic weighted sentence vector is generated; beneficialeffects are as follows. The advantages and disadvantages of semantics, deep learning and machine learning are comprehensively considered; density clustering and convolutional neural network algorithms are applied. Intelligent degree is high, according to the method, the statement with high relevancy with the central content can be quickly extracted to serve as the abstract of the text, various machine learning algorithms are applied to the automatic text abstract to achieve a better abstract effect, the method is possibly the main research direction in future in the field, and in addition, the system according to the invention supplies a tool for automatic extraction of a document abstract based on the method.

The invention discloses a personalized search method for Web service recommendation. The personalized search method comprises the following steps of: 1, preprocessing a WSDL (Web Services Description Language) file, i.e., forming a bag of words through two preprocessing steps of removing stop words and extracting stems; 2, extracting user interest, i.e., calculating weight of each word in the bag of words by using an improved TF-IDF (Term Frequency-Inverse Document Frequency) formula, and multiplying by a time decay factor of the word to obtain a new weight; selecting previous k words according to the weight from large to small as interest words of a user and corresponding weight of each word to form a k-dimension user interest vector; 3, calculating interest similarity, i.e., setting a similarity threshold and selecting the users with interest similarity exceeding the threshold as neighbor users of a target user; and 4, ordering service search results, calculating a recommended predicted value of the service according to similarity of neighbor users and the frequency of selecting service of the users, and arranging the searched results in a descending order according to the recommended predicted value, thereby obtaining the personalized search result.

The invention belongs to the field of data mining and provides a method and system for advertisement recommendation based a microblog. The method comprises the steps that microblog data are read; the microblog data are initialized and a microblog text lexical item set is obtained; stop words of the microblog text lexical item set are deleted and a microblog text original feature lexical item set is obtained; mapping is conducted on the microblog text original feature lexical item set and a feature lexical item dictionary, whether lexical items in the microblog text original feature lexical item set exist in the feature lexical item dictionary or not is judged, and the tf-idf values of the appearing lexical items are calculated and serve as the feature values of the lexical items; whether the lexical items of the feature lexical item dictionary exist in the microblog text original feature lexical item set or not is judged and the feature values of the lexical items which do not appear are marked to be zero; feature vectors of the feature values obtained through calculation are automatically classified to classifications divided in advance; according to an automatic classification result, advertisements are recommended to a user. The advertisements recommended by the method and system are accurate and the effect is good.