21 results about "MinHash" patented technology

The invention discloses a knowledge data de-duplication method and device, a storage medium and computer equipment, relates to the technical field of data processing, is suitable for businesses such as financial science and technology and smart medical treatment, and mainly aims to solve the problem of poor de-duplication effect when large-scale knowledge data is processed in existing knowledge data de-duplication. Comprising the steps of obtaining large-scale knowledge data of related businesses; performing clustering processing on the large-scale knowledge data by adopting a MinHash LSH model to obtain a repeated text clustering result; the repeated text clustering result comprises a plurality of groups of similar data sets; performing word embedding calculation on each group of similar data sets by adopting a bge-m3 model to obtain word embedding corresponding to each group of similar data sets; and respectively carrying out semantic similarity de-duplication processing on the word embedding in each group of similar data sets to obtain a de-duplication result of the knowledge data.

The invention belongs to the technical field of industrial Internet of Things, and particularly discloses a data classification method, system and device based on the industrial Internet of Things and a medium, and the method comprises the steps: obtaining the use authority after the system of the industrial Internet of Things is accessed, and then connecting an industrial robot through an Internet protocol; acquiring equipment parameter data and product detection data based on a data interface of the industrial robot; according to the key features of the product detection data, obtaining a product classification result through a MinHash algorithm; and based on the equipment parameter data and the differential privacy federated learning framework, classifying the data of the industrial robot through the classification model. The invention aims to solve the problem that the data classification method of the industrial Internet of Things in the prior art cannot realize the classification of the data of the industrial robot on the premise of protecting the data privacy and security. The problems that robot management devices in the industrial Internet of Things cannot be efficiently cooperated for joint training, and precise classification of industrial robots cannot be achieved are solved.

The invention discloses an offer information generation method based on a language model, which relates to the field of natural language processing, and comprises the following steps of: retrieving news data from a multi-source corpus, forming an original data set, selecting field information of a title, an abstract, a text, a link, a keyword and a data source from the original data set, and storing the selected field information into a database; carrying out null removal and duplicate removal processing on the title and the text field information, and screening data sources to obtain a to-be-processed data set; on the basis of the to-be-processed data set, text vectorization processing is conducted on the title fields through a language model pre-trained by Transform, sentence vectors are obtained, and a sentence vector set is formed; according to the method, a high-quality to-be-processed data set is constructed through weighted Boolean retrieval and MinHash duplicate removal technologies, in the semantic analysis stage, sentence vectors are generated by adopting a RoBERTa model, unsupervised topic discovery is realized in combination with UMAP dimensionality reduction and HDBSCAN clustering, and the problem of high-dimensional text clustering is effectively solved.

The application discloses a privacy risk detection method, device and equipment of a medical RAG system and a storage medium, wherein a cascade design of a Bloom filter and a MinHash is used to greatly reduce the similarity detection complexity on a large-scale document set, millisecond-level response can be realized under typical hardware conditions, the delay is low, the online medical RAG scene is suitable, not only the text surface similarity is considered, but also medical entities and identifiable information are comprehensively analyzed to form a multi-dimensional risk score, which helps to reduce missed detection and false positives, the detection accuracy is high, the fine-grained risk score and level can be output for each query, resource consumption is controllable, easy to expand, and system integration is friendly.

The invention discloses an approximate repetition detection method and system fusing local retrieval and multi-dimensional decision. The method comprises the steps that a to-be-detected document is preprocessed, and text content and service meta-information are extracted; utilizing MinHash to generate a compact signature; quickly recalling a candidate set from a massive historical library through an LSH index; adopting a paragraph-level weighted Jaccard algorithm to accurately calculate the structural similarity between the candidate document and the document to be detected; when the similarity exceeds a threshold value, intelligent decision making is carried out according to a preset priority chain considering multi-dimensional business rules such as release mechanism levels and time, and finally reserved documents are determined; and recording the structured audit log in the whole process for tracing. The method realizes approximate repetition detection with high recall, high precision, low time consumption and interpretable and auditing decision process, and is especially suitable for massive de-duplication scenes of strong normative texts such as government affair official documents, news announcements and the like.

The application discloses a kind of big model training data deduplication method based on semantic clustering.The method is first filtered by Hash matching and MinHash structure to completely repeated and locally repeated text;Subsequently, a pre-training semantic encoding model is used to generate a deep semantic vector, and a density clustering method is used to construct a semantic cluster, converting global high-complexity comparison into local retrieval within the cluster;Then, combined with the efficient neighbor search based on FAISS, the semantic similar samples within the cluster are grouped and deduplicated;Finally, an information entropy sensing mechanism is introduced, and representative samples are selected according to the content complexity of the samples, to significantly reduce redundancy while maintaining corpus diversity.The application can effectively reduce the risk of model memory caused by data duplication, improve training efficiency, and reduce data processing cost, suitable for various large-scale corpus construction and privacy-sensitive scenarios.

The invention discloses a software similar project screening method based on a similarity hash technology. The method comprises the following steps: selecting a plurality of sample projects to construct a knowledge base; preparing a detected item; reading sample items in the knowledge base, generating a sample item text character string for each sample item, and respectively extracting feature vectors by using Simhash and Minhash; generating a detected item text character string for a detected item, and respectively extracting two feature vectors by using Simhash and Minhash; comparing the Simhash feature vector extracted from the detected item with the Simhash feature vector of each sample item to obtain a corresponding Simhash Hamming distance value; comparing the Minhash feature vector extracted from the detected item with the Minhash feature vector of each sample item to obtain a corresponding Minhash signature similarity value; preliminarily screening out a plurality of groups of candidate items according to a Simhash Hamming distance value and a Minhash signature similarity value; and performing file-level or code snippet-level traceability analysis with finer granularity on each candidate item, and finally screening out the most similar target item.

The invention discloses a similarity query method for polygonal geographic data in a spatial database, and belongs to the technical field of spatial databases and geographic information. The method aims at solving the problem that in the prior art, when mass data are processed, the shape similarity searching efficiency is low. The core of the method comprises the following steps of: firstly, converting each polygon into a fixed-length digital signature with translation and zoom invariance through normalization, gridding and MinHash coding; secondly, the signature is divided into blocks (Band), an index is built, and the signature is stored in a distributed No-SQL database such as Apache Cassandra; and finally, during query, a candidate set is quickly filtered out by using a Band key of a query signature, accurate geometric similarity calculation and sorting are performed on the candidate set, and Top-K most similar polygons are returned. According to the method, efficient and accurate real-time shape similarity query in a large-scale spatial database is realized through a filtering-refining two-stage strategy, and the query efficiency and the system expandability are remarkably improved.

The invention relates to the technical field of artificial intelligence and intelligent information retrieval, and discloses a preparation equipment accessory image efficient retrieval method based on deep hash learning. The method comprises the following steps: constructing a differentiable MinHash layer based on MinHash and differentiable operation, and obtaining an image representation model in combination with a Transform backbone network and a Hash layer; obtaining an image of the industrial accessory, inputting the image into the image representation model, discretizing to obtain a minimum hash code of the image of the industrial accessory, and constructing a bidirectional index in combination with the structured data; obtaining bidirectional indexes of all industrial accessory images and establishing an accessory feature database; the method comprises the steps of obtaining an image of a to-be-queried industrial accessory, inputting the image into an image representation model, discretizing the image to obtain a corresponding minimum hash code, obtaining corresponding structured data on the basis of the minimum hash code, finding the accessory on the basis of the structured data, and completing replacement work. The problem that an existing image learning retrieval method is difficult to identify accessories accurately and efficiently is solved.

The present invention relates to the field of information protection technology, and in particular to an information protection method for supply chain procurement relationship chain data, comprising creating a set of candidate enterprise names; generating a set of candidate enterprise name signatures and a signature of an enterprise name to be evaluated according to a MinHash algorithm, and creating a first obfuscated signature set; generating a signature of a purchasing enterprise name according to a MinHash algorithm, and creating a second obfuscated signature set; combining the candidate enterprise name signatures in the second obfuscated signature set to generate a purchasing enterprise combination, and generating a transaction relationship combination based on the candidate enterprise name signatures in the first obfuscated signature set and the purchasing enterprise combination; sending the transaction relationship combination to a third-party data source, obtaining a credit score for each transaction relationship combination, and selecting a score for the transaction relationship combination of the enterprise to be evaluated and the purchasing enterprise. The present invention overcomes the problem of poor protection effect on enterprise transaction relationship information that needs to be protected in the prior art.

The application provides a URL-based API asset merging method and system, which comprises the following steps: obtaining host access record log data; based on the access record log data, extracting a URL, using a specific separator to split the URI, and forming a word set array; according to the word set data, sequentially taking two words as a relationship, taking the words as nodes, constructing a word relationship graph, and calculating the out-degree and in-degree of the words; according to the out-degree and in-degree obtained in the foregoing step, selecting words with out-degree or in-degree less than a specified threshold value by specifying the threshold value; based on the words obtained in the foregoing step, screening out URIs containing the corresponding words, and calculating the URIs with high similarity in the full-amount URIs by using minhash; based on the URIs selected in the foregoing step, calculating the support degree of the selected corresponding words, and replacing the words with a wildcard string if the support degree is less than a specified threshold value, so as to realize API normalization. The application solves the technical problems of a large API asset list and high dependence on manual operation experience.

The invention discloses a privacy risk detection method, device and equipment for a medical RAG system and a storage medium, which greatly reduce the complexity of similarity detection on a large-scale document set through the cascade design of a Bloom filter and MinHash, can realize millisecond response under typical hardware conditions, is low in delay, is suitable for online medical RAG scenes, and improves the detection efficiency. According to the method, text surface similarity is considered, medical entities and identifiable information are comprehensively analyzed, multi-dimensional risk scores are formed, missing detection and misinformation can be reduced, detection accuracy is high, fine-grained risk scores and grades can be output for each query, resource consumption is controllable, expansion is easy, and system integration is friendly.

The invention provides a weighted hash text duplicate checking method based on semantic weight and template suppression. The method sequentially comprises the following steps: extracting lexical, syntactic and semantic features; calculating word / segment / structure three-level template scores in an online stage based on an offline constructed template dictionary, and carrying out weighted fusion on the word / segment / structure three-level template scores; outputting the feature importance degree by the semantic weighting network subjected to comparative learning training, and combining the feature importance degree with the smooth IDF and the template inhibition factor to form a final weight; according to the weight, generating a weighted SimHash fingerprint or a weighted MinHash fingerprint; the fingerprints are written into a GPU accelerated multi-table LSH index and retrieved, the Hamming distance of the bit fingerprints is calculated, an adaptive threshold value is adopted for judgment, and when and only when the distance is not larger than the threshold value, a near-repetition detection result is output, and a document identifier, the distance and the threshold value are recorded. According to the method, virtual high similarity caused by template content is effectively inhibited, the detection rate and accuracy of anti-plagiarism such as word rewriting are improved, and the method has high efficiency and expandability.

A system and method for an estimation of error bounds for file size calculations using MinHash in deduplication systems. The system includes one or more processors to determine a similarity score between the first file and the second file. The one or more processors are further to determine a size estimation of a combination of the first and second files based on the similarity score. Finally, the one or more processors are to determine a maximum error for the size estimation of the combination of the first and second files, wherein the first and second file are to be combined via deduplication and have at least one shared data segment.

The invention discloses a cross-border data dynamic management and control method and system based on flow analysis, and relates to the technical field of data transmission, and the method comprises the following steps: encrypting a data stream image according to a Hadamard product matrix and a diagonal algorithm of a matrix vector product, carrying out the first verification of a new data stream according to an encryption verification method, and carrying out the second verification of the new data stream; and performing second verification on the new data stream according to the decryption verification method, and judging whether the new data stream is compliant or not based on the first verification result and the second verification result. According to the method, PPMI weighted co-occurrence and sliding step length minimum Hash position are used, a globally unique and sequenced natural number sequence number is generated for each byte graph, a data stream image and a random Hadamard matrix are subjected to Hadamard product, then diagonalization product is carried out, matrix vector diagonal coefficients are written in an encryption process, secondary confirmation independent of a rule base is provided, and an encryption result is obtained. The sequence number sorting space and the encryption matrix are both updated online along with flow distribution, and the corresponding encryption graph template can be quickly issued to the edge node.

The embodiment of the invention discloses a COT data set making method and device. The method comprises the following steps that a set of logic questions which are marked manually and have the accuracy larger than a first preset threshold value is provided to serve as initial seed data; on the basis of the semantic structure and the reasoning chain of the initial seed data, a new topic is generated through a structure transformation mechanism, and the new topic retains original knowledge points of the initial seed data and has a brand new reasoning path and a topic structure; through two-stage filtering of MinHash coarse screening and Jaccard fine screening, questions with the similarity with a historical question bank higher than a second preset threshold value in the new questions are removed; and inputting the new questions into three independent models for logicality verification and parallel solution inspection, screening out the questions which are logically reasonable and have unique solutions through consistency judgment, and incorporating the questions into a COT data set. According to the embodiment of the invention, the generation speed of the COT data set can be increased, and the generation cost of the COT data set can be reduced.

The invention provides a marker molecule subtype division method based on sequence homology, which belongs to the technical field of disease markers, and comprises the following steps of: performing quality screening and standardization processing on an input sequence by establishing a sequence quality evaluation model of a double-layer game structure; a MinHash algorithm is adopted to calculate approximate similarity between sequences, a similarity matrix is generated to reduce calculation complexity, a hierarchical clustering algorithm is adopted to construct a homology tree structure based on the similarity matrix, and a hierarchy center is determined through a dynamic threshold value. A cross center between different levels is established through a cross validation method, weight distribution is optimized by adopting a sequence subtype adaptability model, a special processing center is established for missing data in the comparison process, and the influence of the processing center on a classification result is adjusted through a weight function. And finally, outputting a subtype classification result based on the sequence subtype adaptive model of the Transform architecture. The technical problem that the subtype division accuracy is reduced due to sequence quality difference is solved.

The application relates to the field of artificial intelligence and intelligent information retrieval technology, and discloses a high-efficiency retrieval method for images of ore-dressing equipment accessories based on deep hash learning. The method comprises the following steps: constructing a differentiable MinHash layer based on MinHash and differentiable operation, combining a Transformer backbone network and a hash layer to obtain an image representation model; obtaining the images of industrial accessories, inputting the images into the image representation model, and obtaining the minimum hash code of the industrial accessory images through discretization; combining structured data to construct a bidirectional index; obtaining the bidirectional index of all industrial accessory images and establishing an accessory feature database; obtaining the images of the industrial accessories to be queried, inputting the images into the image representation model, obtaining the corresponding minimum hash code through discretization, obtaining the corresponding structured data based on the minimum hash code, finding the accessories based on the structured data, and completing the replacement work, so that the problem that existing image learning retrieval methods cannot accurately and efficiently identify accessories is solved.

The application provides a verifiable resale attribution and dispute resolution method and system for fine-grained structured data transactions. In the sales stage, a unified data processing specification is established through public universe and three-state normalization processing, different buyers are allocated different column views and bit map bindings are generated, the normalized data is embedded, locally sensitive hash is discretized and minimum hash digest is generated, the buyer binding receipt leaf is organized into an authentication data structure, and the root digest is anchored to the alliance chain smart contract. In the resale stage, the sampling position is determined by the on-chain unpredictable random seed, the authentication path verification and normalization recalculation are executed by the independent committee node to confirm the authenticity of the resale object. In the dispute stage, the suspicious sample is extracted and the disclosure digest is confirmed by the committee threshold, and the dispute object is locked. The proving party generates zero-knowledge proof to compare the coincidence count between the candidate buyer digest and the disclosure digest, and outputs the accusation, exclusion or retention judgment three-state ruling result according to the preset accusation threshold and exclusion threshold. Without disclosing the plaintext delivery data, the application provides a verifiable object authenticity verification channel for legal resale, and a privacy protection attribution mechanism based on similarity detection and double threshold ruling for illegal resale or data leakage.

The invention discloses a text generation method and device, electronic equipment and a storage medium, a lexical item set is generated through preprocessed texts, and the set comprises a plurality of initial texts to be subjected to duplicate removal; calculating a minimum Hash fingerprint of the lexical item set; performing locality sensitive hash binning operation on the initial to-be-deduplicated text through the minimum hash fingerprint to generate a deduplicated candidate set; calculating the text similarity of texts to be subjected to duplicate removal in the duplicate removal candidate set; determining the average length of the to-be-deduplicated text, weighting the text similarity based on the average length, and calculating the fusion similarity of the to-be-deduplicated text; and according to the fusion similarity and a preset similarity threshold, performing a duplicate removal operation on the text to be subjected to duplicate removal, and generating a target text, on the basis of a Minhash-LSH framework, the problem of missed judgment of semantic repetition is effectively solved, high-efficiency, high-precision and self-adaptive duplicate removal of large model training data is realized, and the data quality and security are remarkably improved.