134 results about "Latent Dirichlet allocation" patented technology

Applications of machine learning techniques such as Latent Dirichlet Allocation (LDA) and author-topic models (ATM) to the problems of mining of user roles to specify access control policies from entitlement as well as logs which contain record of the usage of these entitlements are provided. In one aspect, a method for performing role mining given a plurality of users and a plurality of permissions is provided. The method includes the following steps. At least one generative machine learning technique, e.g., LDA, is used to obtain a probability distribution θ for user-to-role assignments and a probability distribution β for role-to-permission assignments. The probability distribution θ for user-to-role assignments and the probability distribution β for role-to-permission assignments are used to produce a final set of roles, including user-to-role assignments and role-to-permission assignments.

The invention discloses a network text segmenting method based on the genetic algorithm, used for segmenting short network texts. The method comprises the following steps of: evaluating a Latent Dirichlet allocation (LDA) model corresponding to a corpus by using a Gibbs sampling method, inferring latent topic information using the model, representing texts by using the latent topic information; then transforming a text-segmenting process into a multi-target optimum process by using a parallel genetic algorithm, and calculating the coherency of segmented units, the divergence among the segmented units and fitness functions by using deeper semantic information; and carrying out the genetic iteration of the text segmenting process, and determining whether the segmenting process terminates based on the similarity among multi-iteration results or the upper limit of iterations to obtain the global optimal solution for segmenting the texts. Therefore, the invention improves the accuracy for segmenting the short network texts.

The invention provides a keyword recommending method and system based on a latent Dirichlet allocation (LDA) model. The keyword recommending method comprises basic LDA training and incremental LDA training. The basic LDA training aims at training texts, probability distribution of basic words to the theme and probability distribution of basic texts to the theme are obtained; the incremental LDA training aims at specific incremental seed words, an incremental LDA model of a training text matched with the incremental seed words is calculated, probability distribution of incremental words to the theme and probability distribution of incremental texts to the theme are obtained, and finally probability distribution of full words to the theme and probability distribution of full texts to the theme are formed. The correlation weight and the final correlation score of any two words in the full model are calculated, and one or more keywords the correlation score of which is highest is recommended. By employing the incremental training model, the theme clustering precision is greatly improved, the theme diversity is increased, and the quality of keywords in the theme is substantially improved.

A method and system applies a probabilistic topic model to content in a tax environment to improve user satisfaction with a question and answer customer support system, according to one embodiment. The probabilistic topic model may be the Latent Dirichlet allocation algorithm or other implementations of probabilistic topic models, according to various embodiments. The method and system include receiving submission content from a user, according to one embodiment. The method and system include applying a probabilistic topic model to the submission content to determine submission content topics and submission content statistics, according to one embodiment. The method and system include generating and providing customer support content at least partially based on the submission content topics and at least partially based on the submission content statistics, to facilitate use of the question and answer customer support system by the user, according to one embodiment.

The invention discloses an automatic annotation method and system for an innovative and creative label on the basis of big data. The method comprises the following steps that: using a Sogou corpus to train Word2vector and LDA (Latent Dirichlet Allocation) to obtain a training result set; carrying out word segmentation on the document data of a user browsing page, removing a stop word, and carrying out word filtering processing; for preprocessed document data, combining an improved TextRank algorithm with the Word2vector to calculate a label from the text data; in addition, carrying out calculation on the preprocessed document through the LDA to obtain the label related to a document data theme; and through a label cloud generation way, realizing visualization, and annotating all text label words in the document data so as to bring convenience for users to read and find key content parts.

The invention relates to a multi-level-point-set characteristic extraction method applicable to ground laser radar point cloud classification. Based on point set characteristics, high-precision classification of four kinds of common ground features including pedestrians, trees, buildings and automobiles and the like in a scene is realized. Firstly, point sets are constructed and a point cloud is re-sampled into a point cloud of different scales and thus point sets which are different in size and provided with layered structures are formed through clustering and characteristics of each point in the point sets are obtained; next, an LDA (Latent Dirichlet Allocation ) method is adopted to synthesizing point-based characteristics of all points in each point set into shape characteristics of the point sets; and at last, based on the shape characteristics of the point set, an Adaboost classifier is adopted to train the point sets of different levels so as to obtain a classification result of the whole point cloud. The multi-level-point-set characteristic extraction method has a higher classification precision and has a classification precision, which is far higher than that of point-based characteristics, Bag-of-Word-based characteristics and characteristics based on probabilistic latent semantic analysis (PLSA), in aspect of pedestrians and vehicles.

The invention discloses a clustering method for network behavior habits based on K-means and LDA (Latent Dirichlet Allocation) two-way authentication. According to the clustering method, webpage properties, keywords and frequency in internet browsing records of persons are utilized to combine with a K-means algorithm, an LDA document topic extracting model and an annealing algorithm. The clustering method comprises the following steps: firstly, performing K-means algorithm clustering and LDA document topic extracting model generation on a staff-label-frequency set and a person browsing record-person-keyword set; secondly, storing and calculating an intermediate result, and then performing K-means and LDA two-way authentication by using the annealing algorithm; calculating a global best topic-classification label sequence, and optimizing a network behavior habit clustering result by taking the global best topic-classification label sequence as a reference. By means of the K-means and LDA two-way authentication, the sensitivity to person-classification labels is improved; by using the annealing algorithm, the optimizing efficiency of the clustering result can be improved, and further the clustering accuracy is improved.

The invention discloses an LDA (latent dirichlet allocation) and VSM (vector space model) based similar Chinese herb literature recommendation method. The method includes: adopting an IKAnalyzer to perform word segmentation on topics and summary information of literature on the basis of a terminological dictionary for Chinese herbs, constructing a vector space, performing dimensionality reduction on the vector space, constructing a semantic dictionary, numbering all lexical items in the dictionary in sequence, performing vectorization through each document on the basis of the semantic dictionary, constructing term vectors of each document, utilizing LDA and a Gibbs sampling algorithm to perform training to obtain probability distribution of each document on themes, then computing a value of similarity between every two documents by the aid of KL divergence, computing cosine similarity of the term vectors of each document on the basis of term frequency, performing joint weighting on the two kinds of similarities prior to performing similarity sorting, and then making recommendation. By the method, the literature, similar both in content and theme, in the Chinese herb literature can be recommended to users, and recommendation results are closer to user requirements.

The invention relates to the field of traditional Chinese medicine information search and discloses a traditional Chinese medicine data mining method based on an LDA (Latent Dirichlet Allocation) topic model. The method comprises the following specific steps: 1) determining two groups of priors, namely prescription-topic and topic-medicament, in the LDA model, and conducting priori assumptions to the two groups of priors by an AS (Asymmetry Symmetry) method, wherein the prescription-topic and the topic-medicament are determined respectively by Alpha and Beta; 2) determining the number of topics in the LDA model; 3) solving the LDA model by a Gibbs sampling method; 4) generating a semantic RDF (Resource Description Framework) document of the LDA model, mapping the result of the LDA model to a tetrad, and expressing the result with the semantic RDF document; 5) associating medicaments with prescriptions to build a prescription-topic-medicament with visual structure network G. The method has the advantages of being suitable for handling and mining a great quantity of traditional Chinese medicine prescriptions and capable of obtaining visual structure models.

The invention discloses a topic model-based judgment document similarity analysis method. A semantic-based semi-automatic and universal similarity analysis method is proposed for judgment documents by adopting an LDA (Latent Dirichlet Allocation) topic model in machine learning. The method mainly comprises the steps of selecting corpora; establishing a similarity tag; performing text preprocessing; performing input selection; performing parameter setting; performing iterative training; generating a model; applying the model; and the like. Based on a general similarity analysis method, the characteristics of rich specialized vocabularies and complex semantics in contents of the judgment documents are fully considered, and the semi-structured characteristics of the judgment documents are utilized, so that the accuracy and applicability of judgment document similarity analysis are improved.

The invention discloses a video semantic representation method and system based on a multi-mode fusion mechanism and a medium. Feature extraction: extracting visual features, voice features, motion features, text features and domain features of a video; Feature fusion: performing feature fusion on the extracted visual, voice, motion and text features and domain features through the constructed multi-level hidden Dirichlet distribution topic model; And feature mapping: mapping the fused features to a high-level semantic space to obtain a fused feature representation sequence. The model utilizesthe unique advantages of the theme model in the semantic analysis field, and the video representation mode obtained through model training on the basis of the model has ideal distinction in the semantic space.

The embodiment of the invention discloses a data processing method, a data processing device and a related device, for increasing the parameter solving speed and the parameter solving accuracy of an hLDA (hierarchical latent Dirichlet allocation) model. The method disclosed by the embodiment of the invention comprises the following steps of: sending the global initial global statistical information to various slave nodes, merging the local statistical information received from the slave nodes to obtain the new global statistical information, calculating the probability distribution between documents and themes and the probability distribution between themes and words according to the new global statistical information if the Gibbs sampling performed by the slave nodes is over, establishing and maximizing a likelihood function of a text set according to the calculated probability distributions to obtain new hLDA hyper-parameters, and calculating and outputting the probability distribution between the documents and the themes and the probability distribution between the themes and the words according to the new hLDA hyper-parameters if the iteration for solving the hLDA hyper-parameters is converged.

The invention discloses a method for expanding a feature space of a short text, comprising the following steps of: (1) selecting an expansion source of the feature space of the short text; (2) preprocessing texts of the text data of the expansion source to obtain a document-word matrix used as a training set; (3) establishing a latent Dirichlet allocation theme model on the document-word matrix of the training set; (4) expressing each short text into a word vector; (5) outputting to obtain underlying theme probability distribution related to the short text by using the word vector of the short text as the input of the latent Dirichlet allocation theme model; (6) expressing a theme of an underlying theme into a theme vector; and (7) combining the theme vector with the word vector together to form the short text with an expanded feature space. The invention expands the feature space of the short text by combining the theme vector determined by the underlying theme with the word vector determined by the short text together and can effectively improve the processing quality of short text information.

The invention provides a data semantic processing method. The data semantic processing method can automatically obtain sematic classified information, form into dynamic analysis processing mode and achieve statement automatic classified processing which faces to a free text transformation format command and is based on semantic information. The data semantic processing method comprises dividing obtained texts to be statements and numbering obtained statements through dividing; constructing an LDA (Latent Dirichlet Allocation) model according to dividing; solving through a Gibbs algorithm to obtain theme distribution of statements and vocabulary distribution of themes in the LDA model; computing theme membership degree values of the statements; confirming categories to which the statements belong according to the theme membership degree values of the statements and outputting category results according to corresponding statement sequence numbers.

Performing semantic analysis on a user-generated text string includes training a neural network model with a plurality of known text strings to obtain a first distributed vector representation of the known text strings and a second distributed vector representation of a plurality of words in the known text strings, computing a relevance matrix of the first and second distributed representations based on a cosine distance between each of the plurality of words and the plurality of known text strings, and performing a latent dirichlet allocation (LDA) operation using the relevance matrix as an input to obtain a distribution of topics associated with the plurality of known text strings.

The invention discloses a parallel data processing method based on the hidden Dirichlet distribution model, which belongs to the data mining field. The method includes three solutions, including the multi-process parallel processing, the multi-thread parallel processing and the composite multi-process multi-thread processing; the data DM for being processed is divided into data segments in equal or unequal length in the three solutions; each data segment is provided with an index; each computer process/thread processes the corresponding data segment through applying the index, so as to obtain the subject information of each data item and generate the local sufficient statistic; when the whole DM is processed, the global sufficient statistic is obtained through the merge of the local sufficient statistics so that the current Mi model is obtained through the estimation until the model becomes convergence. The parallel data processing method can utilize the multi-kernel parallel frame of a single computer and the cluster large-scale parallel capability of multi-computer to realize the high-speed processing of the large-scale text sets and effectively reduce the memory usage during the parallel processing process.

Provided is a tax document hierarchical classification method based on multi-tag classification. Firstly, generated subject distribution is extracted from a latent Dirichlet allocation model, and a latent Dirichlet allocation topic character of a tax file is built; then, tf idf feature vectors corresponding to training data are built, the tf idf feature vectors including the training data and files to be classified are calculated, and similarity is calculated to obtain candidate category tags; finally, source data of candidate category tag nodes are supplemented with auxiliary data, a multi-tag classification model based on transfer learning is built through a transfer learning algorithm TrAdaBoost, and the files to be classified are classified. According to the method, a hierarchical classification problem is converted into a searching stage and a classification stage, calculated amount is greatly reduced by means of incremental candidate category searching, computation complexity is lowered, the tax files are mapped to tax category hierarchical categories by means of the multi-tag classification model based on transfer learning, the auxiliary data are effectively used, and classification performance is improved.

The invention relates to an automatic abstract generation method based on a concept semantic unit. The method comprises the steps that the concept semantic unit is utilized to serve as a carrier of semantic computation and document content representation; a semantic focus of document content is obtained through convergence of document semantic content; automatic generation of a document abstract formed by selecting abstract sentences representing the document content according to the semantic focus is achieved. According to the automatic abstract generation method based on the concept semantic unit, by means of a potential Latent Dirichlet Allocation (LDA) mode, modeling is conducted on a document theme, theme generation treatment is achieved, and the abstract sentences are selected according to the importance degree of the theme. In the computation process, a concept hierarchy network symbol is introduced, merging to semantic information is achieved by means of semantic hierarchy relations in the concept hierarchy network symbol, and the data sparsity problem caused by taking a word as a semantic computation unit is improved.

The application discloses an article classification method and device, computer equipment and a storage medium. The method includes: inputting a word-document matrix, which corresponds to to-be-labeled articles, into a pre-built LDA (Latent Dirichlet Allocation) model to carry out training to obtain a corresponding topic-word matrix and a document-topic matrix; acquiring a topic, which correspondsto each document in the to-be-labeled articles, through the document-topic matrix, and correspondingly adding a topic label for each document to obtain already labeled articles; and classifying the already labeled articles according to the topic labels to obtain article classification results. According to the method, article word segmentation is carried out, then the topic to which each articlebelongs can be obtained simply by acquiring the document-topic matrix, the topics are utilized to classify the articles, automatic learning is adopted to replace manual classification, manpower costsare saved, and classification efficiency is improved.

The invention provides an academic resource acquisition method based on LDA (latent Dirichlet allocation). According to the academic resource acquisition method, a topical crawler is used; an LDA topical model is also used; firstly, a training corpus is provided for the LDA topical model to train to obtain a topical document; the topical crawler further comprises a topical determination module, a similarity calculation module and a URL (Uniform Resource Locator) priority ranking module on the basis of a general network crawler; in the crawling process of the topical crawler, the topical document is adopted to guide calculation of topical similarity, a URL of which the topical similarity is greater than a set threshold is selected, the topical crawler maintains an URL queue of webpages which are not accessed, and the topical crawler sequentially and continuously accesses a webpage of each URL according to a ranking sequence of the URL queue, crawls corresponding academic resources, and continuously stores the crawled academic resources into a database after carrying out classification labeling until the URL in the queue of the webpages which are not accessed is empty; an API (Application Program Interface) of which an academic resource database is open is provided for display and calling; machine learning is fused into the academic resource acquisition method, and academic resource acquisition quality and efficiency are improved.

The invention discloses a legal provision recommendation method based on a LDA (Latent Dirichlet Allocation) topic model. The method comprises the following steps that: extracting a judgment documentset to construct training linguistic data; preprocessing the judgment document; preprocessing a case situation; training the LDA topic model to extract the judgment document set which is similar to the case situation; extracting a recommended legal provision set, designing a legal provision association degree scoring mechanism to calculate an association degree between the legal provision and thecase, and combining with a frequent item set to mine the associated legal provision; and outputting a recommended legal provision list. The step of preprocessing the judgment document comprises the following steps that: extracting the basic situation paragraph and the quoted legal provision list of the case, carrying out Chinese word segmentation on the basic situation of the case, removing legally proprietary stop words, and carrying out legal provision name standardization. By use of the method, a real scene that a judge always looks up similar judgment documents to decide legal provision quotation in a practical judgment process is simulated, the similarity of the judgment document is measured from a semantic level, the similar judgment document can be accurately obtained, associated legal provision recommendation is carried out, and legal provision recommendation accuracy is improved.

The invention discloses a financial public opinion perception method based on a weighted LDA (latent Dirichlet allocation) topic model and belongs to the technical field of natural language understanding and processing as well as network public opinion. Everyday financial public opinions are perceived on the basis of microblog data related to everyday finance and are quantified according to 'everyday financial public opinion comprehensive index'. The 'everyday financial public opinion comprehensive index' is a weighted average of all financial related blog emotion values on the day, and the blog emotion values are a result of text emotion classification of blog content. An SVM (support vector machine) classification model based on weighted LDA is adopted for text emotion classification and adopts the weighted LDA for establishing text represented hidden topic space, objective data indirectly embodying investor sentiment and subjective data directly embodying investor sentiment are organically combined with a new lexical item weight calculation method, and accurate understanding of texts from the semantic level is promoted greatly, so that the text emotion classification effect is better.

The invention discloses a television user family member analysis method based on an LDA (Latent Dirichlet Allocation) model. The method comprises the following steps: extracting information of movies from a database; performing word segmentation on the movie information; taking the segmented words as the input of the LDA model; obtaining each theme of the movies, the distribution probability of the segmented words on each theme and the distribution probability of the movies on each theme through calculation of the LDA model; uniformly dividing 24 hours of a day into a plurality of time periods, and obtaining user feature vectors according to historical records of movies watched by users within different time periods; deducing the member composition of the users according to the distribution probability of the users on each theme, and then analyzing the active time period of each user in a family in combination with the user feature vectors; and when the television is turned on again, finding the corresponding user feature vector according to the time period of the turn-on time, finding the favorite movie theme of the user according to the user feature vector, deducing a family member category, and recommending programs.

The invention relates to a personalized travel package recommendation method based on demand classification and subject analysis. The method includes: analyzing natural language form demands input by a user, utilizing word segmentation, demand classification and other natural language processing techniques to process and classify the user demands so as to obtain rigid demand, flexible demand and negative demand of the user; then utilizing an LDA (latent dirichlet allocation) document theme generation model, making travel service individuals effectively cluster into different service fields by theme similarity, and then conducting similarity matching with the user demands so as to obtain a service list best matching user expectation; finally carrying out travel package design recommendation by means of travel package optimized recommendation algorithm: firstly acquiring a travel package scenic spot set according to user time demand and service priority information; then combining location information, preference information and the like to determine travel package hotel service; and then calculating the optimal journey of every day according to the distance function L, and ranking the travel package according to travel package recommendation index; and finally, selecting travel package catering service according to scenic spot location and user preference. By integrating the processing, the purpose of designing and recommending personalized travel package best meeting the user demand can be realized.

The invention relates to a face and fingerprint-based gender recognition method using a machine vision and pattern recognition technology, and the method comprises the following specific steps: (1) representing image characteristics based on a bag-of-words model, in which a new method for supervising vision word creation is provided to eliminate redundant characteristic number of dimensions and enhance the dimensions helpful for gender classification; (2) putting forward an improved latent Dirichlet allocation (LDA) model to maximize classification intervals, thereby enhancing the recognitioncapability of the entire model; and (3) integrating face and fingerprint visual decisions, training the model separately, and obtaining the final decision through minimizing decision-making risk. Themethod provided by the invention has good robustness and discrimination capacity, and is verified in a large-sized database comprising faces and fingerprints of same persons. Results show that the method can well complete gender recognition, and can be used in all aspects of gender recognition.