14 results about "MinHash" patented technology

The invention discloses an offer information generation method based on a language model, which relates to the field of natural language processing, and comprises the following steps of: retrieving news data from a multi-source corpus, forming an original data set, selecting field information of a title, an abstract, a text, a link, a keyword and a data source from the original data set, and storing the selected field information into a database; carrying out null removal and duplicate removal processing on the title and the text field information, and screening data sources to obtain a to-be-processed data set; on the basis of the to-be-processed data set, text vectorization processing is conducted on the title fields through a language model pre-trained by Transform, sentence vectors are obtained, and a sentence vector set is formed; according to the method, a high-quality to-be-processed data set is constructed through weighted Boolean retrieval and MinHash duplicate removal technologies, in the semantic analysis stage, sentence vectors are generated by adopting a RoBERTa model, unsupervised topic discovery is realized in combination with UMAP dimensionality reduction and HDBSCAN clustering, and the problem of high-dimensional text clustering is effectively solved.

The application discloses a privacy risk detection method, device and equipment of a medical RAG system and a storage medium, wherein a cascade design of a Bloom filter and a MinHash is used to greatly reduce the similarity detection complexity on a large-scale document set, millisecond-level response can be realized under typical hardware conditions, the delay is low, the online medical RAG scene is suitable, not only the text surface similarity is considered, but also medical entities and identifiable information are comprehensively analyzed to form a multi-dimensional risk score, which helps to reduce missed detection and false positives, the detection accuracy is high, the fine-grained risk score and level can be output for each query, resource consumption is controllable, easy to expand, and system integration is friendly.

The invention discloses an approximate repetition detection method and system fusing local retrieval and multi-dimensional decision. The method comprises the steps that a to-be-detected document is preprocessed, and text content and service meta-information are extracted; utilizing MinHash to generate a compact signature; quickly recalling a candidate set from a massive historical library through an LSH index; adopting a paragraph-level weighted Jaccard algorithm to accurately calculate the structural similarity between the candidate document and the document to be detected; when the similarity exceeds a threshold value, intelligent decision making is carried out according to a preset priority chain considering multi-dimensional business rules such as release mechanism levels and time, and finally reserved documents are determined; and recording the structured audit log in the whole process for tracing. The method realizes approximate repetition detection with high recall, high precision, low time consumption and interpretable and auditing decision process, and is especially suitable for massive de-duplication scenes of strong normative texts such as government affair official documents, news announcements and the like.

The application discloses a kind of big model training data deduplication method based on semantic clustering.The method is first filtered by Hash matching and MinHash structure to completely repeated and locally repeated text;Subsequently, a pre-training semantic encoding model is used to generate a deep semantic vector, and a density clustering method is used to construct a semantic cluster, converting global high-complexity comparison into local retrieval within the cluster;Then, combined with the efficient neighbor search based on FAISS, the semantic similar samples within the cluster are grouped and deduplicated;Finally, an information entropy sensing mechanism is introduced, and representative samples are selected according to the content complexity of the samples, to significantly reduce redundancy while maintaining corpus diversity.The application can effectively reduce the risk of model memory caused by data duplication, improve training efficiency, and reduce data processing cost, suitable for various large-scale corpus construction and privacy-sensitive scenarios.

The invention discloses a similarity query method for polygonal geographic data in a spatial database, and belongs to the technical field of spatial databases and geographic information. The method aims at solving the problem that in the prior art, when mass data are processed, the shape similarity searching efficiency is low. The core of the method comprises the following steps of: firstly, converting each polygon into a fixed-length digital signature with translation and zoom invariance through normalization, gridding and MinHash coding; secondly, the signature is divided into blocks (Band), an index is built, and the signature is stored in a distributed No-SQL database such as Apache Cassandra; and finally, during query, a candidate set is quickly filtered out by using a Band key of a query signature, accurate geometric similarity calculation and sorting are performed on the candidate set, and Top-K most similar polygons are returned. According to the method, efficient and accurate real-time shape similarity query in a large-scale spatial database is realized through a filtering-refining two-stage strategy, and the query efficiency and the system expandability are remarkably improved.

The application provides a URL-based API asset merging method and system, which comprises the following steps: obtaining host access record log data; based on the access record log data, extracting a URL, using a specific separator to split the URI, and forming a word set array; according to the word set data, sequentially taking two words as a relationship, taking the words as nodes, constructing a word relationship graph, and calculating the out-degree and in-degree of the words; according to the out-degree and in-degree obtained in the foregoing step, selecting words with out-degree or in-degree less than a specified threshold value by specifying the threshold value; based on the words obtained in the foregoing step, screening out URIs containing the corresponding words, and calculating the URIs with high similarity in the full-amount URIs by using minhash; based on the URIs selected in the foregoing step, calculating the support degree of the selected corresponding words, and replacing the words with a wildcard string if the support degree is less than a specified threshold value, so as to realize API normalization. The application solves the technical problems of a large API asset list and high dependence on manual operation experience.

The invention discloses a privacy risk detection method, device and equipment for a medical RAG system and a storage medium, which greatly reduce the complexity of similarity detection on a large-scale document set through the cascade design of a Bloom filter and MinHash, can realize millisecond response under typical hardware conditions, is low in delay, is suitable for online medical RAG scenes, and improves the detection efficiency. According to the method, text surface similarity is considered, medical entities and identifiable information are comprehensively analyzed, multi-dimensional risk scores are formed, missing detection and misinformation can be reduced, detection accuracy is high, fine-grained risk scores and grades can be output for each query, resource consumption is controllable, expansion is easy, and system integration is friendly.

The invention provides a weighted hash text duplicate checking method based on semantic weight and template suppression. The method sequentially comprises the following steps: extracting lexical, syntactic and semantic features; calculating word / segment / structure three-level template scores in an online stage based on an offline constructed template dictionary, and carrying out weighted fusion on the word / segment / structure three-level template scores; outputting the feature importance degree by the semantic weighting network subjected to comparative learning training, and combining the feature importance degree with the smooth IDF and the template inhibition factor to form a final weight; according to the weight, generating a weighted SimHash fingerprint or a weighted MinHash fingerprint; the fingerprints are written into a GPU accelerated multi-table LSH index and retrieved, the Hamming distance of the bit fingerprints is calculated, an adaptive threshold value is adopted for judgment, and when and only when the distance is not larger than the threshold value, a near-repetition detection result is output, and a document identifier, the distance and the threshold value are recorded. According to the method, virtual high similarity caused by template content is effectively inhibited, the detection rate and accuracy of anti-plagiarism such as word rewriting are improved, and the method has high efficiency and expandability.

A system and method for an estimation of error bounds for file size calculations using MinHash in deduplication systems. The system includes one or more processors to determine a similarity score between the first file and the second file. The one or more processors are further to determine a size estimation of a combination of the first and second files based on the similarity score. Finally, the one or more processors are to determine a maximum error for the size estimation of the combination of the first and second files, wherein the first and second file are to be combined via deduplication and have at least one shared data segment.

The invention discloses a cross-border data dynamic management and control method and system based on flow analysis, and relates to the technical field of data transmission, and the method comprises the following steps: encrypting a data stream image according to a Hadamard product matrix and a diagonal algorithm of a matrix vector product, carrying out the first verification of a new data stream according to an encryption verification method, and carrying out the second verification of the new data stream; and performing second verification on the new data stream according to the decryption verification method, and judging whether the new data stream is compliant or not based on the first verification result and the second verification result. According to the method, PPMI weighted co-occurrence and sliding step length minimum Hash position are used, a globally unique and sequenced natural number sequence number is generated for each byte graph, a data stream image and a random Hadamard matrix are subjected to Hadamard product, then diagonalization product is carried out, matrix vector diagonal coefficients are written in an encryption process, secondary confirmation independent of a rule base is provided, and an encryption result is obtained. The sequence number sorting space and the encryption matrix are both updated online along with flow distribution, and the corresponding encryption graph template can be quickly issued to the edge node.

The application relates to the field of artificial intelligence and intelligent information retrieval technology, and discloses a high-efficiency retrieval method for images of ore-dressing equipment accessories based on deep hash learning. The method comprises the following steps: constructing a differentiable MinHash layer based on MinHash and differentiable operation, combining a Transformer backbone network and a hash layer to obtain an image representation model; obtaining the images of industrial accessories, inputting the images into the image representation model, and obtaining the minimum hash code of the industrial accessory images through discretization; combining structured data to construct a bidirectional index; obtaining the bidirectional index of all industrial accessory images and establishing an accessory feature database; obtaining the images of the industrial accessories to be queried, inputting the images into the image representation model, obtaining the corresponding minimum hash code through discretization, obtaining the corresponding structured data based on the minimum hash code, finding the accessories based on the structured data, and completing the replacement work, so that the problem that existing image learning retrieval methods cannot accurately and efficiently identify accessories is solved.

The application provides a verifiable resale attribution and dispute resolution method and system for fine-grained structured data transactions. In the sales stage, a unified data processing specification is established through public universe and three-state normalization processing, different buyers are allocated different column views and bit map bindings are generated, the normalized data is embedded, locally sensitive hash is discretized and minimum hash digest is generated, the buyer binding receipt leaf is organized into an authentication data structure, and the root digest is anchored to the alliance chain smart contract. In the resale stage, the sampling position is determined by the on-chain unpredictable random seed, the authentication path verification and normalization recalculation are executed by the independent committee node to confirm the authenticity of the resale object. In the dispute stage, the suspicious sample is extracted and the disclosure digest is confirmed by the committee threshold, and the dispute object is locked. The proving party generates zero-knowledge proof to compare the coincidence count between the candidate buyer digest and the disclosure digest, and outputs the accusation, exclusion or retention judgment three-state ruling result according to the preset accusation threshold and exclusion threshold. Without disclosing the plaintext delivery data, the application provides a verifiable object authenticity verification channel for legal resale, and a privacy protection attribution mechanism based on similarity detection and double threshold ruling for illegal resale or data leakage.

The invention discloses a text generation method and device, electronic equipment and a storage medium, a lexical item set is generated through preprocessed texts, and the set comprises a plurality of initial texts to be subjected to duplicate removal; calculating a minimum Hash fingerprint of the lexical item set; performing locality sensitive hash binning operation on the initial to-be-deduplicated text through the minimum hash fingerprint to generate a deduplicated candidate set; calculating the text similarity of texts to be subjected to duplicate removal in the duplicate removal candidate set; determining the average length of the to-be-deduplicated text, weighting the text similarity based on the average length, and calculating the fusion similarity of the to-be-deduplicated text; and according to the fusion similarity and a preset similarity threshold, performing a duplicate removal operation on the text to be subjected to duplicate removal, and generating a target text, on the basis of a Minhash-LSH framework, the problem of missed judgment of semantic repetition is effectively solved, high-efficiency, high-precision and self-adaptive duplicate removal of large model training data is realized, and the data quality and security are remarkably improved.