75 results about "Sentence length" patented technology

The invention discloses an online traditional Chinese medicine text named entity identifying method based on deep learning. The method includes the steps that online traditional Chinese medicine text data are obtained through a web crawler, and named entities of the obtained online traditional Chinese medicine text data are labeled with existing terminological dictionaries and human assistance; a word2vec tool is used for carrying out learning on large-scale label-free linguistic data, and word vectors with fixed length are obtained and used for forming a corresponding glossary; word segmentation is carried out on the online traditional Chinese medicine text data, words are converted into the word vectors with the fixed length by searching for the glossary, the word vectors serve as input of a convolutional neural network, and a blank character is used for filling when sentence length is insufficient; output of the convolutional neural network serves as input of a bidirectional long-short-time memory recurrent neural network, and an identification result of the online traditional Chinese medicine text data words to be identified is output. Compared with a traditional method for named entity identifying, the method reduces complexity and workload of feature extraction, simplifies the processing process and remarkably improves identification efficiency.

A confidence scoring system can include a model trained using features extracted from translations that have received user translation ratings. The features can include, e.g. sentence length, an amount of out-of-vocabulary or rare words, language model probability scores of the source or translation, or a semantic similarity between the source and a translation. Parameters of the confidence model can then be adjusted based on a comparison of the confidence model output and user translation ratings, where the user translation ratings can be selected or weighted based on a determination of individual user fluentness. After the confidence model has been trained, it can produce confidence scores for new translations. If a confidence score is higher than a threshold, it can indicate the translation should be selected for automatic presentation to users. If the confidence score is below another threshold, it can indicate the translation should be updated.

The invention discloses a filtering method of a bilingual corpus and the method comprises the following steps: A. ratio flag value of sentence length of English-Chinese bilingual sentence pair is confirmed; B. the number of different parts of speech in the English-Chinese bilingual sentence pair is respectively counted, the matching number of the corresponding words in a bilingual intertranslating dictionary and words of the part of speech are calculated and the interpretation eigenvalue is confirmed according to the number of different parts of speech and the matching number; C. the filtration and classification are carried out by the ratio eigenvalue of the sentence length and the interpretation eigenvalue according to a classification model established by using a training set in advance. The invention discloses a bilingual corpus system; the invention also provides a filtering method of the bilingual corpus and a system thereof, which are used for improving universality, accuracy and recalling rate of the corpus.

The invention discloses a text similarity calculation method and device, and relates to the technical field of text processing, which can accurately calculate the similarity between texts in a text with complex expression. The method comprises the steps of obtaining training word segmentation corpora obtained after word segmentation is conducted on text corpora with different sentence lengths; inputting the training word segmentation corpora as training data into a supervision model for training, and constructing a sentence vector conversion model which is used for converting sentences in the text corpus into sentence vectors for representing text characteristics; adjusting characteristic parameters in the sentence vector conversion model according to the sentence vector which is obtained by training and represents the text characteristics; based on the adjusted sentence vector conversion model, performing sentence vector conversion on the plurality of target texts to obtain a plurality of sentence vectors representing the characteristics of the target texts; and calculating the similarity among the plurality of target texts according to the plurality of sentence vectors representing the characteristics of the target texts.

The invention discloses an automatic generating method and device for Internet headlines. The method comprises the following steps: dividing the main body of a piece of news, keeping sentences with the length within a preset length range, and marking the sentences as kept sentences; respectively computing the Similarity (s) between each kept sentence and a headline of the news and the Weight (s) of each kept sentence; computing the rank score of each kept sentence according to a formula that Rank (s) is equal to Weight (s)/Similarity (s); selecting the kept sentence with the highest rank score as the headline, wherein Rank (s) is the rank score of the kept sentence. The method and the device automatically recognize a sentence which can best reflect the value of a piece of news, and take the sentence as the headline of the news.

The invention provides a Song ci poetry text message hiding technology based on hybrid encryption, which belongs to the information hiding and data security directions in the field of computers. The Song ci poetry text message hiding technology comprises the steps of encrypting secret information to be hidden by using an advanced encryption standard (AES) in a hybrid manner, encrypting an AES secret key by using an elliptic curve cryptography (ECC) algorithm, passing all information after encryption processing through a 140 tune name template library of the complete collection of Song ci poetry, and hiding the information by means of the system which is composed of templates, a dictionary, a steganographic device and an extractor, wherein the system can generate steganographic Song ci poetry through a random selection or template designation method according to the length of a cryptograph, and the sentence length, grammatical style and intonation sentence pattern of the steganographic Song ci poetry conform to the original Song ci poetry completely, thereby achieving the purposes of obfuscating attackers and ensuring secure transmission of the hidden information. The Song ci poetry text message hiding technology disclosed by the invention can solve the security problem of data transmission in channels, can provide double security measures of information hiding and data encryption, and has high practical application value.

The invention discloses a method for grading Chinese electronic document reading on the Internet, comprising firstly determining the frequency distributions of Chinese characters, word groups and sentence structure indexes in different grades of documents; selecting the Chinese characters and the word groups for grading document reading, and avoiding the interference of often-used words and little-used words, then analyzing the word composition of a to-be-graded target document, analyzing the document to be a two-tuple vector (of words and occurrence number); calculating the sentence structure indexes of the document comprising an average paragraph length, an average sentence length, the length difference between the longest sentence and the shortest sentence and the like; and finally using the Naive Bayes method for determining the reading grade of the document based on the word composition information and the sentence structure information of the Chinese document. The reading grade of a Chinese electronic document is efficiently determined by analyzing the Chinese characters and word group composition of the document, combining with the sentence structures of the document, reasoning from the frequency distribution of each word and the structure indexes in different reading grades of documents and applying the Naive Bayes method.

An information handling system is disclosed for generating tags of a file including a document or a webpage posting. The generating tags of a file include converting a webpage posting to a PDF document. The method further includes extracting tags provided by users. The method includes scanning the extracted data from a glossary PDF document to identify keywords of the glossary PDF document in accordance with a sentence length. The method further includes extracting data from the PDF document and scanning the extracted data to identify keywords of the PDF document in accordance with a sentence length. The method further includes reapplying selected keywords to the tags of the file.

The invention discloses a word segmentation method based on Bi-LSTM. The method includes the steps that training corpus data is converted into character-level corpus data; the corpus data is segmentedaccording to sentence length, and several sentences are obtained; then the obtained sentences are grouped according to the sentence length, and a data set comprising n groups of sentences is obtained; several pieces of data are extracted from the data set to serve as iteration data; the each-time iteration data is converted into a fixed-length vector to be sent into a depth learning model Bi-LSTM, and parameters of the depth learning model Bi-LSTM are trained; when the loss value iteration change generated by the depth learning model is smaller than a set threshold value and is not decreasedany more or the maximum iteration time number is reached, training of the depth learning model is terminated, and the trained depth learning model Bi-LSTM is obtained; corpus data to be predicted is converted into character-level corpus data, the character-level corpus data is sent into the trained depth learning model Bi-LSTM, and a word segmentation result is obtained.

A method for instance-based sentence boundary determination optimizes a set of criteria based on examples in a corpus, and provides a general domain-independent framework for the task by balancing a comprehensive set of sentence complexity and quality constraints. The characteristics and style of naturally occurring sentences are simulated through the use of semantic grouping and sentence length distribution. The method is parameterized so that it is easily adapts to suit a Natural Language Generation (NLG) system's generation.

The invention discloses an LSTM-based word segmentation method. The method comprises the steps of 1) converting training corpus data into character-level corpus data; 2) dividing the corpus data according to a sentence length to obtain multiple sentences, and according to the sentence length, grouping the obtained sentences to obtain a data set comprising n groups of sentences; 3) extracting multiple pieces of data from the data set to serve as iterative data; 4) converting the iterative data each time into a fixed-length vector, inputting the vector to a deep learning model LSTM, training parameters of the deep learning model LSTM, and when a loss value iterative change generated by the deep learning model is smaller than a set threshold, is no longer reduced or reaches a maximum iterative frequency, stopping the training of the deep learning model, and obtaining a trained deep learning model LSTM; and 5) converting to-be-predicted corpus data into the character-level corpus data, andinputting the character-level corpus data to the trained deep learning model LSTM to obtain a word segmentation result.

The invention discloses an LSTM-CNN-based word segmentation method. The method comprises the steps of converting training corpus data into character-level corpus data; performing statistics on characters of the corpus data to obtain a character set and numbering the characters to obtain a character number set; performing statistics on character tags to obtain a tag set, and numbering the tags to obtain a tag number set; dividing corpora according to sentence lengths, and grouping obtained sentences according to the sentence lengths to obtain a data set comprising n groups of sentences; randomly selecting a sentence group from the data set without replacement, wherein the characters of each sentence form a piece of data w, and the corresponding tag set is y; converting the data w into the corresponding number and tag y to be input to a LSTM-CNN of a model, and training parameters of the deep learning model; and converting to-be-predicted data into data matched with the deep learning model, and inputting the data to the trained deep learning model to obtain a word segmentation result.

The invention relates to a question and answer processing method and device, computer equipment and a readable storage medium, and the method comprises the steps: obtaining a to-be-answered question, and determining a similar question similar to the to-be-answered question from a question and answer database; converting the similar problem into a similar problem vector; based on any two or more of a literal similarity score, a semantic similarity score, a sentence editing distance score, a sentence length score, a knowledge base similarity score, a hash similarity score, a word vector similarity score, a keyword score, a keyword editing distance score and an entity word editing distance score and a similar problem vector, determining a target question matched with the to-be-answered question from the similar questions; and determining a target answer corresponding to the target question from the question and answer database.

The invention relates to a semantic role recognition method based on a phrase structure tree. The method comprises the steps of sentence pruning, wherein when a system inputs one sentence, phrase analysis is performed on the sentence, the analyzed result is subjected to pruning through a parenthesis or a coordination structure, the complexity of the sentence is simplified, and the length of the sentence is shortened; clause extracting processing, wherein on the basis of the phrase structure tree, clauses in the pruned sentences are extracted, the extracted clauses and the remaining portion obtained after the clauses are extracted are subjected to semantic role analysis separately, the complete sentence semantic role is obtained, and the analysis result of the semantic role is reduced; boundary correction, wherein the reduced semantic role is combined with the phrase tree to perform predicate argument boundary correction on the sentences, and finally the sentence semantic role analysisresult is output. The sentence complexity is simplified, the sentence length is shortened, the complex and long sentence can be effectively processed, and the semantic role labeling condition is improved.

The invention relates to the field of natural language processing, and provides an example sentence searching method according to query. The method comprises the following steps of: obtaining the query input by a user; processing the query input by the user; searching sample sentences matched with the query in an example sentence library, and calculating the relativity of the query and the example sentences; carrying out example sentence relativity scoring adjustment according to a usage diversity or translation diversity principle, and sorting the example sentences; outputting the example sentences and presenting phrases in the example sentences. The invention further provides an example sentence searching system according to the query. According to the scheme provided by the invention, various factors are comprehensively considered in calculation of the relativity of the query and the example sentences, and specifically, the features of the related phrases to the query in the example sentences, the syntactic features, the example sentence structure integrality feature, the sentence length feature and the digital noise feature of punctuations in the example sentences are comprehensively considered for calculating the relativity of the query and the example sentences; and the method is superior to other relativity calculation methods.

The invention discloses a text splicing method and a device thereof. The method comprises the following steps of: obtaining a to-be-spliced current text fragment, obtaining an average sentence lengthcorresponding to the current text fragment, obtaining a first semantic score, in a semantic model, of the current text fragment, and obtaining a second semantic score, in the semantic model, of a candidate sentence comprising the current text fragment, wherein the current text fragment is a starting fragment of the candidate sentence; and splicing the current text fragment according to the averagesentence length, the first semantic score and the second semantic score, so as to obtain a target sentence corresponding to the current text fragment. During sentence segmentation, sentence lengths are considered, so that the sentence lengths are proper, long difficult sentences or massive short sentences are avoided, and the sentence lengths are relatively stable; and scoring of the semantic model is also considered, so that the sentence segmentation correctness can be improved, semantic meanings of the sentences are not damaged and the intelligibility of the sentences is improved.

A method is disclosed that utilizes specific techniques, based upon empirical study, to significantly increase the ability of a trainee to remain focused on the training materials and subject matter and actually learn and retain the training subject matter. More specifically, the present invention utilizes audio and/or visual (e.g., a personal computer) elements, with a strict set of rules which must be followed regarding sentence length, narrators, and underlying music within the dialog to create a specific rhythmic “feel” to the training. As a result of using such techniques, significantly improved results over prior art training methods can be obtained.

The invention discloses a translation model optimization method for dynamically adjusting length punishment and translation length. The method comprises the steps of obtaining standard data in a specified language direction as a standard bilingual data set for various index prediction; performing word segmentation operation on the standard bilingual data set, and performing further training to obtain a new training data set; modifying a neural machine translation model decoder part, and automatically predicting the optimal length punishment value of the current sentence pair; performing lengthstatistics to obtain a target statement sub-length; preparing an independent feedforward neural network model so that a translation finally predicted by the model tends to a translation result with the optimal length; and enabling the Transformer neural machine translation model to dynamically adjust the length penalty and the optimal translation sentence length for different sentences. Accordingto the method, the length punishment and the dynamic adjustment of the translation length in the model translation process are realized, the realization is simple, the method is effective, the practicability is high, and the model translation quality improvement effect is obvious.

The invention discloses a mixed-corpus word segmentation method based on Bi-LSTM. The method includes the steps that training mixed corpus data is converted into character-level corpus data; characters of the corpus data are subjected to statistics, a character set is obtained, the characters are numbered, and a character number set is obtained; labels of the characters are subjected to statistics, a label set is obtained, the labels are numbered, and a label number set is obtained; corpus is segmented according to sentence length, obtained sentences are grouped according to the sentence length, and a data set is obtained; a sentence group is randomly selected from the data set without releasing, multiple sentences are extracted from the sentence group, the characters of each sentence formdata w, and the corresponding label set is y; the data w is converted into corresponding numbers and labels y to be sent into a model Bi-LSTM, and parameters of the depth learning model Bi-LSTM are trained; data to be predicted is converted into data matched with the depth learning model, the data is sent into the trained depth learning model Bi-LSTM, and a word segmentation result is obtained.

The invention discloses a mixed corpus word segment method based on an LSTM (Long Short Term Memory)-CNN (Convolutional Neural Network). The method comprises the steps of converting training mixed corpus data into the mixed corpus data at a character level; counting the mixed corpus data characters to obtain a character set, and numbering each character to obtain a character serial number set; counting character labels to obtain a label set, numbering the labels to obtain a label serial number set; segmenting the corpus according to a sentence length, and grouping the obtained sentences according to the sentence length to obtain a data set; randomly selecting a sentence subgroup from the data set, extracting a plurality of sentences from the sentence subgroup, wherein the characters of each sentence form a datum w, and a corresponding label set is y; converting the datum w into a corresponding serial number and sending the label y to a model LSTM-CNN, and training a parameter of a deeplearning model; and converting to-be-predicted mixed corpus data into data matched with the deep learning model, sending the to-be-predicted mixed corpus data to the trained deep learning model to obtain a word segment result.

The invention discloses a corpus processing method. The corpus processing method comprises the steps: acquiring the phoneme frequency of each phoneme and the sentence length frequency of each sentence in an original corpus, wherein the phoneme frequency of each phoneme represents the number of the same phonemes in the original corpus, and the sentence length frequency of each sentence represents the number of sentences with the same sentence length in the original corpus; and calculating a frequency parameter of each sentence according to the phoneme frequency and the sentence length frequency, and taking the frequency parameter as a score of the sentence, wherein the frequency parameter is in negative correlation with the phoneme frequency, and is in negative correlation with the sentence length frequency. The invention further discloses a corpus processing device and a storage medium. According to the corpus processing method, a reliable standard is provided for corpus selection, so that the reliability of corpus sentence selection during screening can be improved, and the screening efficiency of a large number of text corpora is effectively improved, and the corpus processing method is suitable for large-scale corpus information screening tasks.