32 results about "MinHash" patented technology

Methods, systems and computer program product embodiments for hashing techniques for determining similarity between data sets are described herein. A method embodiment includes, initializing a random number generator with a weighted min-hash value as a seed, wherein the weighted min-hash value approximates a similarity distance between data sets. A number of bits in the weighted min-hash value is determined by uniformly sampling an integer bit value using the random number generator. A system embodiment includes a repository configured to store a plurality of data sets and a hash generator configured to generate weighted min-hash values from the data sets. The system further includes a similarity determiner configured to determine a similarity between the data sets.

The invention discloses a multi-label learning design method based on a hashing method. Through the combination of a hashing algorithm and a multi-label learning algorithm based on Bayesian statistics, the correlation between labels is effectively utilized so as to improve the predicting performance of a multi-label learning model, labels and neighbors of the labels are introduced to computation of the posterior probability through the characteristics of the neighbors, the correlation between the labels is fully considered, and the accuracy of the algorithms is improved; the problem that the label space in multi-label learning of large-scale data is higher in dimension and sparse is solved through an MinHash algorithm; the purpose of learning large-scale data is achieved by finding the neighbors through locality sensitive hashing (LSH), the neighbors can be rapidly and efficiently found, and the expandability of the multi-label learning algorithm is improved.

The invention discloses a method for detecting repetition data of social media. The method comprises following steps: dividing each text data of social data into multiple text elements constituting sets corresponding to text data; utilizing a Hush function to map all text elements in sets to corresponding Hash values and obtaining minimum Hash values, repeating mapping for multiple times in order to obtain an array composed of multiple minimum Hash values as the minimum Hash signature for text data; utilizing a locality-sensitive hashing algorithm to map text elements of each minimum Hash value to different detection queues; and calculating Jaccard similarity between any two text elements in the same detection queue. Text elements with Jaccard similarity larger than threshold value are determined as repetition data.The a method for detecting repetition data of social media is capable of increasing repeatability detection efficiency of large texts.

The invention discloses a method and device for obtaining similar object set and providing similar object set. The method comprises as follows: obtaining input file comprising M objects, N attributes, attribute values corresponding to each attribute; inputting each attribute to first level of pre-created minimum hash function minhash, obtaining the returned value of the first level of minhash of each attribute; according to each attribute, weighted value corresponding to the attribute in the current object and the second level of pre-created minhash function, obtaining the returned value of the second level of the minhash of each attribute; calculating the combined minhash value of each attribute in each object respectively; determining the minimum value of the combined minhash value corresponding to each attribute of the same object as the minhash value of the object; circularly executing the operation to each object for K times, respectively obtaining K minhash values in allusion to each object; inputting K minhash values of each object to the locality sensitive hashing (LSH) computing framework. The method and device are capable of improving the operating efficiency, and improving the validity and accuracy degree of the similar object information.

The invention discloses a set similarity calculation method and system based on minhash. The method includes the steps that each element in a set is mapped into a first hash value with an m-bit length through a hash function, and 2k class groups are established, wherein each class group corresponds to one tag which is a second hash value with a k-bit length, and different class groups correspond to different tags; for any set, the first hash values corresponding to the elements in the set are distributed into the class groups corresponding to the tags with the same first k bits as the first hash values; minhash values, corresponding to the class groups, of the set are determined according to the distribution result; the minhash values, corresponding to the class groups, of the set form an array serving as a minhash signature of the set; according to the minhash signatures of any two sets, the similarity of the two sets is calculated. By means of the technical scheme, the minhash signature speed can be greatly increased, and thus the set similarity calculation speed is greatly increased.

The invention provides a method and a system for identifying homologous binary files in a database. The database comprises multiple binary basic files. The method comprises the steps of obtaining signatures of to-be-identified files and signatures of the basic files according to a min-hash algorithm; for any signature, performing bucket dividing processing on the signature according to a bucket dividing method; according to a reverse indexing method and the signatures, subjected to bucket dividing, of all the basic files, obtaining dictionaries in one-to-one correspondence with buckets, wherein each dictionary comprises at least one key value pair; according to character strings in the buckets of the to-be-identified files, traversing the corresponding dictionaries, and according to valuescorresponding to matching keys, obtaining the homologous binary files of the to-be-identified files. According to the method and the system, the signatures are obtained by adopting the min-hash algorithm and the bucket dividing is performed by adopting a local sensitive hash algorithm, so that the calculation amount can be remarkably reduced; and by adopting the reverse indexing method, an indextable is established for all the signatures, so that the speed of identifying the homologous binary files is increased.

The invention discloses a method and a device for generating text fingerprint information. The method comprises the steps as follows: an initial feature vector of a text is extracted; a weighted value of at least one element in the initial feature vector is endowed with a multiple value of the minimum weighted value; the weighted values of the other elements are endowed with the minimum weighted value; the corresponding element is added to the initial feature vector according to the multiple to form a new feature vector; and the text fingerprint information is generated after minimum hash algorithm is carried out on the new feature vector. The method and the device for generating the text fingerprint information disclosed by the invention can improve the accuracy of fingerprint information, so that information clustering has a relatively excellent effect.

The invention relates to an MES-oriented mass data redundancy elimination method and system. According to the MES-oriented mass data redundancy removal method and system, a minimum hash algorithm is adopted to compress preprocessed data to obtain a minimum hash signature, an LSH (Local Sensitive Hash) algorithm is adopted to avoid similarity calculation, and the data is subjected to bucket division according to hash values, so the time complexity in the process of finding out similar repeated data from mass data is greatly reduced, and the overall efficiency of data processing is improved. Moreover, the Jaccard similarity is used as a screening condition, the data of which the Jaccard similarity is greater than a threshold value is defined as potential similar data, and then similarity detection from distribution to overall is performed on the potential similar data to remove similar repeated data, so the redundancy removal capability is improved.

The invention relates to a combination optimizing method based on a Lucene index section, and belongs to the technical field of the computer index. The method comprises the following steps: combiningcurrent node load information and section information of index, building a combination analyzing module to judge whether to meet a combination condition or not; according to a dictionary file contained in each index section, to obtain a characteristic matrix in the index with respect to an index section, processing by combining a minHash algorithm and a minimum hash signature algorithm, so as to calculate the signature matrix of the index section; through combining the signature matrix of the index section and a Jaccard similarity principle, calculating a similarity coefficient between the index sections, and according to the similarity coefficient, dividing the index sections into different similar sets; and using a similarity evaluation model to grade each similar set, and sorting according to a set score, selecting one or more sets with the highest score to be combined by a combination thread. The optimizing method is capable of reducing the effect of combination operation to performance of an index function and a search function and effectively improving a search speed.

The invention provides a cloud storage similar data detection method and system based on meta-semantic embedding. The method comprises the following steps: carrying out CDC partitioning on all data in a cloud storage data domain; extracting feature vectors of all the CDC blocks by adopting a MinHash algorithm; processing the context feature vector of any CDC block based on a Mask algorithm, and inputting all the processed context feature vectors into a neural network model for training to obtain a meta-semantic model of a cloud storage data field; extracting semantic feature vectors of the new data uploaded to the cloud storage data domain; and inputting the semantic feature vector of the new data into the new neural network model initialized by the meta-semantic model for similarity detection. According to the method, full-text semantics are embedded based on a meta-semantic embedding method, the reliability of data feature extraction is enhanced, repeated training of the neural network is avoided, and therefore the calculation overhead is reduced.

The invention discloses a malicious file detection method and device, computer equipment and a storage medium, relates to the technical field of information, and mainly aims at relieving the calculation pressure of malicious file detection. The method comprises the steps of obtaining a calling interface sequence corresponding to a to-be-detected file; determining a feature sequence corresponding to the calling interface sequence, and determining a minimum hash signature corresponding to the to-be-detected file according to the feature sequence; according to the minimum Hash signature corresponding to the to-be-detected file, distributing the to-be-detected file to corresponding Hash buckets under different Hash intervals, wherein a plurality of Hash buckets exist in any Hash interval; determining a first target sample file which is distributed to the same hash bucket with the to-be-detected file in the different hash intervals; and judging whether the to-be-detected file is a malicious file or not according to the category information corresponding to the first target sample file. The method and the device are suitable for malicious file detection.

The invention discloses a document detection processing method and device, a storage medium and electronic equipment. The method comprises the following steps: receiving a document detection request sent by a client; performing text processing and hash table generation processing on the read document content in the to-be-detected document to obtain a first hash signature; searching a minimum Hash signature list in a pre-stored document fingerprint database according to the first Hash signature to obtain an index value of a second Hash signature having coincident elements with the first Hash signature; and positioning a sample document corresponding to the second hash signature through the index value, and calculating a similarity value between the first hash signature and the second hash signature to obtain an overlap ratio value between the to-be-detected document and the sample document. The technical problems that in the prior art, a document detection processing method cannot recognize extraction conditions of a small number of documents, and the document overlap ratio cannot be calculated are solved.

The invention discloses an intelligent agent behavior responsibility investigation method based on social network privacy negotiation system, which realizes agent behavior responsibility investigation through qualitative responsibility investigation and quantitative responsibility investigation processes, and adopts a forward simulation negotiation process and a reverse reproduction negotiation process in the qualitative responsibility investigation process. And whether the privacy negotiation agent has improper behaviors or not is accurately judged, and the specific occurrence position of the privacy negotiation agent is accurately locked when the improper behaviors exist. Three quantitative responsibility investigation methods including a simple quantification method, a weighted mahalanobis distance method and an improved Minhash method are further provided, the responsibility quantification value of the privacy negotiation agent can be obtained, and therefore the severity degree of improper behaviors is quantified. According to the invention, the problems of untrusted, unsafe and malicious behaviors of the intelligent agent in the current social network privacy negotiation system are solved.

The invention provides a similar crowd extension algorithm based on a locality sensitive Hash algorithm. An open source tool datasketch is adopted to calculate original data features so as to obtain weighted minimum Hash of feature vectors of all users, the calculation amount can be greatly reduced, the calculation speed and accuracy are improved, and meanwhile the calculation cost is reduced; and in addition, a local sensitive Hash model constructed by using an open source tool datasketch can be obtained according to the memory size of the memory and the requirement of calculation accuracy, so that the accuracy of the local sensitive Hash model is high.

A set sorting music recommendation method for user implicit feedback data comprises the steps of obtaining user-song interaction data, then finding out a nearest neighbor of a user waiting for recommendation songs from all users of music software by adopting a MinHash-based locality sensitive hashing algorithm, then mining potential preference songs of the user waiting for recommendation songs according to a collaborative filtering algorithm, dividing all songs into three types: heard songs, potential preference songs and remaining songs; and sequencing the listened songs, the potential preference songs and the remaining songs, generating a personalized music recommendation list, and completing music recommendation. According to the method, the cooperative relation between the users can be utilized, the sequence relation between the song sets is considered, and compared with a traditional method, a better music recommendation effect is achieved.