119 results about "Document similarity" patented technology

A document similarity detection and classification system is presented. The system employs a case-based method of classifying electronically distributed documents in which content chunks of an unclassified document are compared to the sets of content chunks comprising each of a set of previously classified sample documents in order to determine a highest level of resemblance between an unclassified document and any of a set of previously classified documents. The sample documents have been manually reviewed and annotated to distinguish document classifications and to distinguish significant content chunks from insignificant content chunks. These annotations are used in the similarity comparison process. If a significant resemblance level exceeding a predetermined threshold is detected, the classification of the most significantly resembling sample document is assigned to the unclassified document. Sample documents may be acquired to build and maintain a repository of sample documents by detecting unclassified documents that are similar to other unclassified documents and subjecting at least some similar documents to a manual review and classification process. In a preferred embodiment the invention may be used to classify email messages in support of a message filtering or classification objective.

A device, computer program product and a method for searching, navigating or retrieving documents in a set of electronic documents, including performing a link analysis of the set of electronic documents. The link analysis includes one of analyzing at least two of the set of documents with at least a portion of a similarity graph constructed among the set of documents and analyzing the at least two of the set of documents with the at least a portion of the similarity graph and at least a portion of a hyperlink graph constructed from hyperlinks between the set of documents. Also described is a method for building a similarity matrix.

One embodiment of the present invention provides a system for optimizing and customizing document-similarity calculation. During operation, the system presents a collection of similar documents to a user, collects feedback on the similarity of the documents from the user, generates generic rules for calculating document similarity, and filters documents with customized similarity calculation based on the feedback provided by the user.

Described herein are methods for finding substantially similar/different sources (files and documents), and estimating similarity or difference between given sources. Similarity and difference may be found across a variety of formats. Sources may be in one or more languages such that similarity and difference may be found across any number and types of languages. A variety of characteristics may be used to arrive at an overall measure of similarity or difference including determining or identifying syntactic roles, semantic roles and semantic classes in reference to sources.

The invention discloses a document citation network visualization and document recommendation method and system, and relates to the fields of document influence analysis and information visualization. The method comprises the following steps of calculating the importance degree of documents according to authors, time information, citation numbers and other inherent attributes, document similarities and transfer values generated by introducing behavior quantitative analysis, and sorting the documents; then performing clustering on the sorted documents and performing visualization on the clustered result to establish a dual-layer network model, and displaying important documents in a clear manner; and finally, recommending the documents in the center of clustering displayed in the visualization manner to a user. The document citation network visualization and document recommendation method and system are high in usability and can help science researchers to rapidly screen out most authoritative papers.

A text semantic analysis method and system can realize semantic analysis of text data base on lexical level and sentence level. Aiming at the semantic analysis at the lexical level, the invention firstly adopts an improved word segmentation algorithm to solve the problem that English words are segmented only by spaces. Secondly, based on word segmentation, TF-IDF modeling is performed to obtain weight value; Then the text is vectorized by weighting and summing the weight value and the word vector trained by Word2Vec, and finally the document similarity is solved. At the same time, the invention considers the contribution degree of the vocabulary to the document content and the semantic status to calculate the similarity degree of the document, the result has higher accuracy, and provide agood foundation for subsequent text clustering. The present invention extracts subject-predicate object structure based on text segmentation, part-of-speech tagging, syntactic analysis and dependencyrelation for sentence level semantic analysis. The invention realizes the extraction of subject-predicate-object structures of various sentence types in all aspects, and realizes the noun expansion function, which is more consistent with the manual extraction result.

An object of the present invention is to provide a document processing device and document processing method that can provide a search result satisfactory to a user with respect to WWW documents in which a number of links among WWW documents is low and a number of accesses by users is low. An access pattern collection unit 101 generates an access user vector u_j of one WWW document D_j and an access user vector u_je of another document D_je. A user similarity computing unit 105 computes a document similarity sim (u_j, u_je) which indicates a user similarity between the WWW document D_j and WWW document D_je. A keyword vector smoothing unit 106 acquires a smoothed keyword weight vector w′_j by correcting a keyword weight vector w_j in one document, using the computed document similarity sim (u_j, u_je). An rearranging unit 110 calculates an evaluation value B_SCORE for input information for searching, based on the smoothed keyword weight vector w′_j.

The invention discloses a topic model-based judgment document similarity analysis method. A semantic-based semi-automatic and universal similarity analysis method is proposed for judgment documents by adopting an LDA (Latent Dirichlet Allocation) topic model in machine learning. The method mainly comprises the steps of selecting corpora; establishing a similarity tag; performing text preprocessing; performing input selection; performing parameter setting; performing iterative training; generating a model; applying the model; and the like. Based on a general similarity analysis method, the characteristics of rich specialized vocabularies and complex semantics in contents of the judgment documents are fully considered, and the semi-structured characteristics of the judgment documents are utilized, so that the accuracy and applicability of judgment document similarity analysis are improved.

Systems and methods are provided to compute indicators of completeness of the work output of a transformation of text-based content, worker capacity in performing the transformation, and/or the degree of matching between a unit of work and a worker, based on information collected about complexity of works, times and throughput of workers, rating of work outputs and using natural language processing techniques and machine learning techniques, such as language detection, longest common substring, length ratio, document similarity, etc. The indicators are utilized to optimize job pickup and output submission for online crowdsourcing tasks related to transformation of text-based content, such as transcription, translation, proofreading, etc.

The invention relates to a document similarity calculation method and a near-duplicate document detection method and device. The calculation method comprises the following steps: performing word segmentation processing on two documents to be detected to obtain respective participle sets of the documents to be detected; calculating the edition similarity of all participle pairs in the two participle sets, wherein two participles in each participle pair come from the two participle sets respectively; establishing sides among the participle pairs of which the edition similarity meets a certain requirement in all the participle pairs to obtain a weighted biograph, wherein the edition similarity is the weights of the sides of corresponding participle pairs; calculating the maximum weighted matching value of the weighted bi-graph; calculating the similarity between the documents to be detected by using the maximum weighted matching value. By adopting the document similarity calculation method and the near-duplicate document detection method and device provided by the invention, high accuracy is achieved, near-duplicate texts comprising participle set edition errors can be identified effectively, the near-duplicate document detection accuracy is increased, the calculation complexity is lowered, and the calculation efficiency is optimized.

A system and method for measuring the similarity between SVG documents. The present invention involves reducing the respective documents to their minimal logical representations and then analyzing the representations using tree isomorphism techniques. Applications can then use this comparison data to more efficiently perform actions such as content compression, content streaming and content searching.

An embodiment of the invention provides a detection method and system, relates to the field of a data processing technology, and solves the problem that the conventional approximate duplicated document detection method cannot meet higher requirements in terms of a precision ratio and a recall ratio. In the embodiment of the invention, a method combining multi-feature fingerprint inquiry and document similarity comparison is adopted, multi-feature fingerprints can accurately reflect discriminative features of a web page document to be detected and other web page documents, and records in accordance with conditions can be rapidly inquired according to a corresponding relation of the feature fingerprints and approximate duplicated documents in an existing data base, so that the accuracy rate and the efficiency of approximate duplicated document detection can be improved. With the adoption of the detection method of the document similarity comparison, the situation that a web page document to be detected surely belongs to an approximate duplicated document but cannot be detected by the multi-feature fingerprint inquiry due to the fact that the data base is defective can be prevented, so that the recall ration of the approximate duplicated document detection is improved.

To perform multi-pattern searching, a preprocessing engine populates a SUFFIX table, a PREFIX table and a PATTERN table. The SUFFIX table combines data conventionally stored in SHIFT and HASH tables. Pointers in the SUFFIX table refer to corresponding segments in the PREFIX table. Each PREFIX table segment is sorted by a prefix hash. A PATTERN table includes a hash of each full pattern sorted and grouped into segments, with each segment corresponding to a suffix hash and prefix hash combination. Pointers in the PREFIX table refer to corresponding segments in the PATTERN table. The PREFIX and PATTERN can be kept in secondary storage, allowing potentially billions of patterns to be used. After preprocessing, patterns are evaluated against a source file. A document metric is determine to qualitatively describe the similarity between the source file and each pattern file.

An input section inputs a document set. A normalization section calculates a similarity as a relative value between documents, with respect to combinations of a plurality of documents, in the document set. The normalization section employs the tf.idf method. In tf.idf method, a document vector and a significance of a word included in the document is used to perform normalization to convert each similarity to an absolute value.

A method for measuring file similarity based on file structure includes finding out subsubject sequence of each file from X and Y files to be compared by utilizing file structure analysis means, utilizing similarity measurement means to calculate similarity value between two subsubjects of different files, setting up weighted bipartite graph G={ X,Y, E } on obtained subsubject sequence and similarity value, solving out optimum match for weighted bipartite graph and carrying out normalization treatment on total weight value of optimum match so as to obtain similarity value of X and Y files.

The invention provides a judgement document similarity calculation method, a judgement document similarity search device, and computer equipment. The judgement document similarity calculation method comprises the following steps of: obtaining at least two judgement documents; extracting judgement keywords of one or more defendants in each judgement document; and determining the similarity between corresponding judgement documents according to the similarity between the judgement keywords corresponding to the defendants in different judgement documents. According to the judgement document similarity calculation method, the judgement document similarity calculation correctness is improved.

The invention provides a fast document type recognition method based on full-sized feature extraction. The method comprises the following steps of document image preprocessing, including image zooming, graying and noise filtering; document image feature extraction, including Hessian matrix building, scale space generation, primary determination of feature points, precise positioning of the feature points, main direction determination of selected feature points, feature point descriptor construction and feature value string generation; document image feature value comparison, including document similarity calculation and comparison algorithm optimization. According to the method, an image is preprocessed through software, and additional hardware equipment does not need to be added. According to the method, the scale-invariant feature is creatively introduced for improving a typical SURF feature extraction algorithm, so that the problem of matching failure due to error amplification of the SURF algorithm caused by scale variations is fundamentally solved. The method has the advantage that a multi-thread technology and a large cache are used for solving the problems of large data volume calculation during comparison and the harsh time requirement of a user on an electronic government affair platform.

The invention discloses a document similarity calculation method facing cloud storage based on a fully homomorphic password technology. The method comprises the following steps that a data owner uploads a document ID, an encrypted document cryptograph and a cryptograph of a document Hash value to a cloud server; a public key certificate is issued to a cloud service provider and a data user; the data user encrypts a simhash value of a document with a similarity to be calculated and uploads to the cloud service provider; the cloud service provider carries out fully homomorphic addition operation of a document simhash cryptograph value to be calculated and a data owner document simhash cryptograph value and returns an operation result to the data owner; the data owner acquires Hamming distances between the documents and returns a document ID with a top distance sorting to the cloud service provider. By using the method, the calculation is performed under a cryptograph state. During a calculation process, any information related to the document is not revealed to the cloud service provider and other attackers so that a data secret of the data owner and a query data secret of the data user are protected.

The invention provides a large-scale document similarity detection method. The method comprises the steps of S1, calculating the similarity of other information of documents in a document set; S2, enabling each document content to correspond to a signature S and a f-dimensional vector V; S3, performing word segmentation processing on the document content; S4, comprehensively calculating a weight of a feature word x; S5, mapping the feature word x into a signature h by using a hash function, traversing all bits of the h, and adjusting the V; S6, traversing the V, adjusting the signature S, andfinally generating a signature value, corresponding to the document content, of the signature S; S7, dividing the signature value corresponding to the document content into n blocks, mapping the blocks to a bucket by using the hash function, and judging whether double hash is performed or not; S8, taking the documents of the same bucket as a candidate pair, and calculating the similarity; and S9,judging whether the documents are similar documents or not. The method is high in detection accuracy and high in executive efficiency, and can be widely used in the internet large-scale data mining.

The invention discloses a method for automatically classifying documents based on a K nearest neighbor algorithm under a power cloud environment. By the method, a MapReduce programming framework of cloud calculation is improved; a Map function finishes the calculation of document similarity; and a reduce function defines K samples having highest similarity, counts weights of various classifications where the nearest neighbor belongs and outputs the classification having the largest weight so as to automatically classify the documents. By the method, the task for classifying a large quantity of documents can be finished quickly, so the execution time of the task for classifying the documents is shortened greatly, and the classifying efficiency is improved; and the method has robustness.

The invention relates to a news and case similarity calculation method based on an asymmetric twin network, and belongs to the technical field of natural language processing. The method comprises thefollowing steps: firstly, selecting a sentence representation document most relevant to a news title by calculating the similarity between sentences and titles in a text so as to remove redundant sentences in the news text; describing and modeling a document and a case by using an asymmetric twin network, fusing the case element serving as supervision information into the asymmetric twin network to encode a news document and case description in consideration of key semantic information of the case contained in the case element, and finally judging the correlation between news and the case by calculating document similarity. According to the method, similarity calculation is carried out on the news text and the case description based on the asymmetric twin network, semantic coding modelingcan be carried out on the unbalanced news text and case description, and the accuracy of similarity calculation can be improved.