54 results about "MinHash" patented technology

The present invention discloses a method, device, and medium for identifying software component analysis, which relates to the technical field of software technology. Specifically, it includes: obtaining all files in the software to be identified to obtain a first data set; extracting meta-file data to obtain a second data set, and obtaining a first open-source file set through first feature matching; performing deletion processing on the first data set to obtain a third data set, calculating the minimum hash signature through the program abstract syntax tree and the corresponding sub-syntax tree to generate a second feature set, and obtaining a second open-source file set through third feature matching; performing deletion processing on the third data set to obtain a fourth data set, obtaining an updated syntax tree based on the abstract syntax tree and the fourth data set and constructing a control flow diagram, and obtaining a third open-source file set through fourth feature matching between the third feature set and a preset reference database; merging the first open-source file set, the second open-source file set, and the second open-source file set to obtain the total open-source component set in the software to be identified.

The invention discloses a knowledge data de-duplication method and device, a storage medium and computer equipment, relates to the technical field of data processing, is suitable for businesses such as financial science and technology and smart medical treatment, and mainly aims to solve the problem of poor de-duplication effect when large-scale knowledge data is processed in existing knowledge data de-duplication. Comprising the steps of obtaining large-scale knowledge data of related businesses; performing clustering processing on the large-scale knowledge data by adopting a MinHash LSH model to obtain a repeated text clustering result; the repeated text clustering result comprises a plurality of groups of similar data sets; performing word embedding calculation on each group of similar data sets by adopting a bge-m3 model to obtain word embedding corresponding to each group of similar data sets; and respectively carrying out semantic similarity de-duplication processing on the word embedding in each group of similar data sets to obtain a de-duplication result of the knowledge data.

The invention discloses a system and a method for converting a natural language into an SQL (Structured Query Language) based on contents and annotations, and relates to the field of databases. Vectorizing the annotations of the tables and the annotations of the fields, and storing the vectorized annotations into a vector database; the content of the table is stored in a local file after being subjected to minimum hashing; extracting keywords from the natural language of the user through a large language model; vectorizing the extracted keywords, and comparing the vectorized keywords with data in a vector database to obtain a first similar result; performing minimum hashing on the extracted keyword, and comparing the keyword with a local file to obtain a second similar result; assembling the first similar result and the second similar result, and selecting a table through a large language model; based on the selected table, obtaining a table building statement, and generating an SQL statement through a large language model; and based on the query database selected by the user, executing the SQL statement to obtain data. The method has good universality, and the accuracy of data query is improved.

The invention belongs to the technical field of industrial Internet of Things, and particularly discloses a data classification method, system and device based on the industrial Internet of Things and a medium, and the method comprises the steps: obtaining the use authority after the system of the industrial Internet of Things is accessed, and then connecting an industrial robot through an Internet protocol; acquiring equipment parameter data and product detection data based on a data interface of the industrial robot; according to the key features of the product detection data, obtaining a product classification result through a MinHash algorithm; and based on the equipment parameter data and the differential privacy federated learning framework, classifying the data of the industrial robot through the classification model. The invention aims to solve the problem that the data classification method of the industrial Internet of Things in the prior art cannot realize the classification of the data of the industrial robot on the premise of protecting the data privacy and security. The problems that robot management devices in the industrial Internet of Things cannot be efficiently cooperated for joint training, and precise classification of industrial robots cannot be achieved are solved.

Strings of a text file representing a configuration of a target device are respectively tokenized into tokens for the configuration. The tokens for the configuration are shingled. A target device signature representing the configuration of the target device is generated by applying a min-wise independent permutations locality sensitive hashing (MinHash) technique to the tokens as have been shingled. Whether the configuration of the target device is anomalous is identified based on the target device signature.

The invention discloses an offer information generation method based on a language model, which relates to the field of natural language processing, and comprises the following steps of: retrieving news data from a multi-source corpus, forming an original data set, selecting field information of a title, an abstract, a text, a link, a keyword and a data source from the original data set, and storing the selected field information into a database; carrying out null removal and duplicate removal processing on the title and the text field information, and screening data sources to obtain a to-be-processed data set; on the basis of the to-be-processed data set, text vectorization processing is conducted on the title fields through a language model pre-trained by Transform, sentence vectors are obtained, and a sentence vector set is formed; according to the method, a high-quality to-be-processed data set is constructed through weighted Boolean retrieval and MinHash duplicate removal technologies, in the semantic analysis stage, sentence vectors are generated by adopting a RoBERTa model, unsupervised topic discovery is realized in combination with UMAP dimensionality reduction and HDBSCAN clustering, and the problem of high-dimensional text clustering is effectively solved.

The application discloses a privacy risk detection method, device and equipment of a medical RAG system and a storage medium, wherein a cascade design of a Bloom filter and a MinHash is used to greatly reduce the similarity detection complexity on a large-scale document set, millisecond-level response can be realized under typical hardware conditions, the delay is low, the online medical RAG scene is suitable, not only the text surface similarity is considered, but also medical entities and identifiable information are comprehensively analyzed to form a multi-dimensional risk score, which helps to reduce missed detection and false positives, the detection accuracy is high, the fine-grained risk score and level can be output for each query, resource consumption is controllable, easy to expand, and system integration is friendly.

The invention relates to the technical field of biological feature recognition, in particular to a face template protection method and system based on Hadamard transform and improved MinHash, a registered user face template is generated through the face template protection method, and the registered user face template is stored in a protected face template database. According to the face template protection method, a face template is generated through face image feature extraction, local Hadamard transformation and an improved MinHash algorithm; the method comprises the following steps: acquiring a face image of a to-be-authenticated user, generating a target face template of the to-be-authenticated user through a face template protection method, and performing similarity matching on the target face template and a registered user face template in a protected face template database to judge whether the face image of the to-be-authenticated user is a registered user face image or not. According to the method, two-layer protection can be realized on the face template in face recognition, the safety is relatively high, the recognition accuracy and generation efficiency of the face template are effectively improved, and the method has a relatively good application prospect in the field of face recognition identity authentication and authorization.

The invention discloses an approximate repetition detection method and system fusing local retrieval and multi-dimensional decision. The method comprises the steps that a to-be-detected document is preprocessed, and text content and service meta-information are extracted; utilizing MinHash to generate a compact signature; quickly recalling a candidate set from a massive historical library through an LSH index; adopting a paragraph-level weighted Jaccard algorithm to accurately calculate the structural similarity between the candidate document and the document to be detected; when the similarity exceeds a threshold value, intelligent decision making is carried out according to a preset priority chain considering multi-dimensional business rules such as release mechanism levels and time, and finally reserved documents are determined; and recording the structured audit log in the whole process for tracing. The method realizes approximate repetition detection with high recall, high precision, low time consumption and interpretable and auditing decision process, and is especially suitable for massive de-duplication scenes of strong normative texts such as government affair official documents, news announcements and the like.

The present invention claims protection for an efficient fault-tolerant dynamic phrase search method based on forward and backward privacy, which includes a data owner, a cloud server and a user. The method includes the following steps: a key generation phase, generating keys and public parameters. In the index construction phase, the data owner designs an index structure using piecewise linear chaotic mapping, a minhash function cascaded with an AND‑OR structure, a Bloom filter, an inverted index and a matrix. In the trapdoor generation phase, the user generates a trapdoor using a phrase. In the phrase search phase, the cloud server checks the stored encrypted index and returns an intermediate encryption result. After receiving it, the user decrypts it and returns the obtained file identifier to the cloud server. The cloud server returns the encrypted file in response, and the user decrypts the file. In the update token generation phase, the user generates a token and sends it to the cloud server. In the index update phase, the cloud server updates the index and documents according to the token. The present invention achieves forward privacy, backward privacy, efficient search and dynamic update.

The invention provides an ancient Chinese language punctuation prediction method and system, electronic equipment and a medium. The ancient Chinese language punctuation prediction method comprises the steps of obtaining training data; carrying out splitting processing on the training data, and obtaining a training data set by utilizing split data blocks; constructing an index database by using a minimum hash algorithm to obtain a reference text of the training data set; training an initial language model by using the reference text and an original text of the training data set to obtain an ancient language punctuation prediction model; and predicting a to-be-predicted text by using the ancient language punctuation prediction model to obtain a prediction result. According to the ancient language punctuation prediction method, the problem of missing report of continuous punctuation prediction can be avoided, and the accuracy of ancient language punctuation prediction is improved.

The application discloses a kind of big model training data deduplication method based on semantic clustering.The method is first filtered by Hash matching and MinHash structure to completely repeated and locally repeated text;Subsequently, a pre-training semantic encoding model is used to generate a deep semantic vector, and a density clustering method is used to construct a semantic cluster, converting global high-complexity comparison into local retrieval within the cluster;Then, combined with the efficient neighbor search based on FAISS, the semantic similar samples within the cluster are grouped and deduplicated;Finally, an information entropy sensing mechanism is introduced, and representative samples are selected according to the content complexity of the samples, to significantly reduce redundancy while maintaining corpus diversity.The application can effectively reduce the risk of model memory caused by data duplication, improve training efficiency, and reduce data processing cost, suitable for various large-scale corpus construction and privacy-sensitive scenarios.

The invention discloses a software similar project screening method based on a similarity hash technology. The method comprises the following steps: selecting a plurality of sample projects to construct a knowledge base; preparing a detected item; reading sample items in the knowledge base, generating a sample item text character string for each sample item, and respectively extracting feature vectors by using Simhash and Minhash; generating a detected item text character string for a detected item, and respectively extracting two feature vectors by using Simhash and Minhash; comparing the Simhash feature vector extracted from the detected item with the Simhash feature vector of each sample item to obtain a corresponding Simhash Hamming distance value; comparing the Minhash feature vector extracted from the detected item with the Minhash feature vector of each sample item to obtain a corresponding Minhash signature similarity value; preliminarily screening out a plurality of groups of candidate items according to a Simhash Hamming distance value and a Minhash signature similarity value; and performing file-level or code snippet-level traceability analysis with finer granularity on each candidate item, and finally screening out the most similar target item.

The invention discloses a similarity query method for polygonal geographic data in a spatial database, and belongs to the technical field of spatial databases and geographic information. The method aims at solving the problem that in the prior art, when mass data are processed, the shape similarity searching efficiency is low. The core of the method comprises the following steps of: firstly, converting each polygon into a fixed-length digital signature with translation and zoom invariance through normalization, gridding and MinHash coding; secondly, the signature is divided into blocks (Band), an index is built, and the signature is stored in a distributed No-SQL database such as Apache Cassandra; and finally, during query, a candidate set is quickly filtered out by using a Band key of a query signature, accurate geometric similarity calculation and sorting are performed on the candidate set, and Top-K most similar polygons are returned. According to the method, efficient and accurate real-time shape similarity query in a large-scale spatial database is realized through a filtering-refining two-stage strategy, and the query efficiency and the system expandability are remarkably improved.

The invention relates to the technical field of artificial intelligence and intelligent information retrieval, and discloses a preparation equipment accessory image efficient retrieval method based on deep hash learning. The method comprises the following steps: constructing a differentiable MinHash layer based on MinHash and differentiable operation, and obtaining an image representation model in combination with a Transform backbone network and a Hash layer; obtaining an image of the industrial accessory, inputting the image into the image representation model, discretizing to obtain a minimum hash code of the image of the industrial accessory, and constructing a bidirectional index in combination with the structured data; obtaining bidirectional indexes of all industrial accessory images and establishing an accessory feature database; the method comprises the steps of obtaining an image of a to-be-queried industrial accessory, inputting the image into an image representation model, discretizing the image to obtain a corresponding minimum hash code, obtaining corresponding structured data on the basis of the minimum hash code, finding the accessory on the basis of the structured data, and completing replacement work. The problem that an existing image learning retrieval method is difficult to identify accessories accurately and efficiently is solved.

The present invention relates to the field of information protection technology, and in particular to an information protection method for supply chain procurement relationship chain data, comprising creating a set of candidate enterprise names; generating a set of candidate enterprise name signatures and a signature of an enterprise name to be evaluated according to a MinHash algorithm, and creating a first obfuscated signature set; generating a signature of a purchasing enterprise name according to a MinHash algorithm, and creating a second obfuscated signature set; combining the candidate enterprise name signatures in the second obfuscated signature set to generate a purchasing enterprise combination, and generating a transaction relationship combination based on the candidate enterprise name signatures in the first obfuscated signature set and the purchasing enterprise combination; sending the transaction relationship combination to a third-party data source, obtaining a credit score for each transaction relationship combination, and selecting a score for the transaction relationship combination of the enterprise to be evaluated and the purchasing enterprise. The present invention overcomes the problem of poor protection effect on enterprise transaction relationship information that needs to be protected in the prior art.

The invention relates to the technical field of spatio-temporal data mining, in particular to a massive trajectory hierarchical parallel clustering method and system based on spatial similarity, and the method comprises the steps: carrying out the grid cell division of a trajectory region according to obtained trajectory data; converting the trajectory data into set data based on the divided grid units; according to a set form of the set data, calculating a MinHash signature corresponding to each track, and forming a signature matrix by the MinHash signatures of all tracks; dividing the obtained signature matrix into a plurality of bands, and mapping tracks in the bands into buckets; classifying the tracks mapped into the same bucket in at least one band into the same class; according to the method, under the conditions that the number of classes in the trajectory data set does not need to be known in advance and the trajectory data set does not need to be trained, the trajectory is adaptively and rapidly divided into a plurality of classes according to the spatial similarity of the trajectory.

The application provides a URL-based API asset merging method and system, which comprises the following steps: obtaining host access record log data; based on the access record log data, extracting a URL, using a specific separator to split the URI, and forming a word set array; according to the word set data, sequentially taking two words as a relationship, taking the words as nodes, constructing a word relationship graph, and calculating the out-degree and in-degree of the words; according to the out-degree and in-degree obtained in the foregoing step, selecting words with out-degree or in-degree less than a specified threshold value by specifying the threshold value; based on the words obtained in the foregoing step, screening out URIs containing the corresponding words, and calculating the URIs with high similarity in the full-amount URIs by using minhash; based on the URIs selected in the foregoing step, calculating the support degree of the selected corresponding words, and replacing the words with a wildcard string if the support degree is less than a specified threshold value, so as to realize API normalization. The application solves the technical problems of a large API asset list and high dependence on manual operation experience.

The invention discloses an identification software component analysis method and device, and a medium, and relates to the technical field of software technology, and the method specifically comprises the following steps: obtaining all files in to-be-identified software to obtain a first data set; extracting meta-file data to obtain a second data set, and obtaining a first open source file set through first feature matching; performing deletion processing on the first data set to obtain a third data set, performing calculation through a program abstract syntax tree and a corresponding sub-syntax tree to obtain a minimum hash signature and generate a second feature set, and performing third feature matching to obtain a second open source file set; performing deletion processing on the third data set to obtain a fourth data set, obtaining an updated syntax number tree based on the abstract syntax tree and the fourth data set, performing construction to obtain a control flow chart, and performing fourth feature matching through a third feature set and a preset reference database to obtain a third open source file set; and combining the first open source file set, the second open source file set and the second open source file set to obtain a total set of open source components in the to-be-identified software.

The invention discloses a privacy risk detection method, device and equipment for a medical RAG system and a storage medium, which greatly reduce the complexity of similarity detection on a large-scale document set through the cascade design of a Bloom filter and MinHash, can realize millisecond response under typical hardware conditions, is low in delay, is suitable for online medical RAG scenes, and improves the detection efficiency. According to the method, text surface similarity is considered, medical entities and identifiable information are comprehensively analyzed, multi-dimensional risk scores are formed, missing detection and misinformation can be reduced, detection accuracy is high, fine-grained risk scores and grades can be output for each query, resource consumption is controllable, expansion is easy, and system integration is friendly.

The invention provides a weighted hash text duplicate checking method based on semantic weight and template suppression. The method sequentially comprises the following steps: extracting lexical, syntactic and semantic features; calculating word / segment / structure three-level template scores in an online stage based on an offline constructed template dictionary, and carrying out weighted fusion on the word / segment / structure three-level template scores; outputting the feature importance degree by the semantic weighting network subjected to comparative learning training, and combining the feature importance degree with the smooth IDF and the template inhibition factor to form a final weight; according to the weight, generating a weighted SimHash fingerprint or a weighted MinHash fingerprint; the fingerprints are written into a GPU accelerated multi-table LSH index and retrieved, the Hamming distance of the bit fingerprints is calculated, an adaptive threshold value is adopted for judgment, and when and only when the distance is not larger than the threshold value, a near-repetition detection result is output, and a document identifier, the distance and the threshold value are recorded. According to the method, virtual high similarity caused by template content is effectively inhibited, the detection rate and accuracy of anti-plagiarism such as word rewriting are improved, and the method has high efficiency and expandability.

The present invention relates to a retrieval method, device, storage medium and terminal based on an improved MinHash algorithm. The retrieval method includes: S1. Download m binary component libraries of different types and different versions to obtain a component set S; S2. Extract features for each component in the component set S respectively; S3. Calculate the weight value of the eigenvalue through the feature weight assignment module, and calculate the component hash value through the MinHash algorithm. If the minimum hash value selects the feature f i , while recording its hash value, associate its feature weight value w(f i ); S4. Then calculate the similarity between two components through the similarity calculation formula. The present invention can more accurately identify the similarity of different components and has a wider adaptability.

In example embodiments, techniques are provided to detect LOLBin attacks using a trained machine learning model that classifies command lines as benign or malicious. The machine learning model may be trained using a dataset of command line data that describes executed binary executable files, sourced from the log of events of compute instances. The dataset may be sampled using an approximate content-based logarithmic sampling algorithm (e.g., an algorithm that employs logarithmic sampling based on a locality sensitive hash, for example, a MinHash). The dataset may be labeled and featurized. The featurized labeled dataset may be used to train the machine learning model, which is then deployed to detect LOLBin attacks on a compute instance. In response to detection of a LOLBin attack, a remedial action may be performed on the compute instance.

A system and method for an estimation of error bounds for file size calculations using MinHash in deduplication systems. The system includes one or more processors to determine a similarity score between the first file and the second file. The one or more processors are further to determine a size estimation of a combination of the first and second files based on the similarity score. Finally, the one or more processors are to determine a maximum error for the size estimation of the combination of the first and second files, wherein the first and second file are to be combined via deduplication and have at least one shared data segment.

The present application provides a method, system, electronic device and medium for predicting ancient Chinese punctuation, the method comprising: obtaining training data; splitting the training data and obtaining a training data set using the split data blocks; constructing an index library using a minimum hash algorithm to obtain a reference text of the training data set; training an initial language model using the reference text and the original text of the training data set to obtain an ancient Chinese punctuation prediction model; and predicting a text to be predicted using the ancient Chinese punctuation prediction model to obtain a prediction result. This method for predicting ancient Chinese punctuation can avoid the problem of underreporting in continuous punctuation prediction and improve the accuracy of ancient Chinese punctuation prediction.

The invention belongs to the technical field of protein sequence clustering, and provides a large-scale protein sequence clustering method, system and product based on minimum hashing. The technical scheme is to construct a unique ID for each sequence, and construct a k-mer set of the sequence; use the same hash function to map the k-mers in the k-mer set into hash values ​​one by one, and select the minimum M hash values ​​as the minhash set of the sequence; combine the minhash set of the sequence and the set grouping rule to group the protein sequence; wherein the set grouping rule is that if there is a collision between the minhash sets of two sequences, they are divided into the same group; cluster each group of sequences separately, and integrate the clustered results to obtain the final clustering result. The time overhead of clustering is greatly reduced.

The invention discloses a cross-border data dynamic management and control method and system based on flow analysis, and relates to the technical field of data transmission, and the method comprises the following steps: encrypting a data stream image according to a Hadamard product matrix and a diagonal algorithm of a matrix vector product, carrying out the first verification of a new data stream according to an encryption verification method, and carrying out the second verification of the new data stream; and performing second verification on the new data stream according to the decryption verification method, and judging whether the new data stream is compliant or not based on the first verification result and the second verification result. According to the method, PPMI weighted co-occurrence and sliding step length minimum Hash position are used, a globally unique and sequenced natural number sequence number is generated for each byte graph, a data stream image and a random Hadamard matrix are subjected to Hadamard product, then diagonalization product is carried out, matrix vector diagonal coefficients are written in an encryption process, secondary confirmation independent of a rule base is provided, and an encryption result is obtained. The sequence number sorting space and the encryption matrix are both updated online along with flow distribution, and the corresponding encryption graph template can be quickly issued to the edge node.

The invention relates to the technical field of information protection, in particular to an information protection method for supply chain purchase relation chain data, which comprises the following steps: creating an alternative enterprise name set; according to a MinHash algorithm, generating an alternative enterprise name signature set and a to-be-evaluated enterprise name signature, and creating a first confusion signature set; generating a buyer enterprise name signature according to a MinHash algorithm, and creating a second confusion signature set; combining the alternative enterprise name signatures in the second confusion signature set to generate a purchaser enterprise combination, and generating a transaction relation combination according to the alternative enterprise name signatures in the first confusion signature set and the purchaser enterprise combination; the transaction relation combinations are sent to a third-party data source, the credit score of each transaction relation combination is obtained, the score of the transaction relation combination of the to-be-evaluated enterprise and the buyer enterprise is selected, and the problem that in the prior art, the protection effect on enterprise transaction relation information needing to be protected is poor is solved.

The embodiment of the invention discloses a COT data set making method and device. The method comprises the following steps that a set of logic questions which are marked manually and have the accuracy larger than a first preset threshold value is provided to serve as initial seed data; on the basis of the semantic structure and the reasoning chain of the initial seed data, a new topic is generated through a structure transformation mechanism, and the new topic retains original knowledge points of the initial seed data and has a brand new reasoning path and a topic structure; through two-stage filtering of MinHash coarse screening and Jaccard fine screening, questions with the similarity with a historical question bank higher than a second preset threshold value in the new questions are removed; and inputting the new questions into three independent models for logicality verification and parallel solution inspection, screening out the questions which are logically reasonable and have unique solutions through consistency judgment, and incorporating the questions into a COT data set. According to the embodiment of the invention, the generation speed of the COT data set can be increased, and the generation cost of the COT data set can be reduced.

The invention provides a marker molecule subtype division method based on sequence homology, which belongs to the technical field of disease markers, and comprises the following steps of: performing quality screening and standardization processing on an input sequence by establishing a sequence quality evaluation model of a double-layer game structure; a MinHash algorithm is adopted to calculate approximate similarity between sequences, a similarity matrix is generated to reduce calculation complexity, a hierarchical clustering algorithm is adopted to construct a homology tree structure based on the similarity matrix, and a hierarchy center is determined through a dynamic threshold value. A cross center between different levels is established through a cross validation method, weight distribution is optimized by adopting a sequence subtype adaptability model, a special processing center is established for missing data in the comparison process, and the influence of the processing center on a classification result is adjusted through a weight function. And finally, outputting a subtype classification result based on the sequence subtype adaptive model of the Transform architecture. The technical problem that the subtype division accuracy is reduced due to sequence quality difference is solved.