367 results about "Bloom filter" patented technology

The invention relates to the technical field of information retrieval, and discloses a distributed storage and indexing method, which comprises the following steps of: dynamically fragmenting an original file through an improved consistent Hash algorithm to generate a plurality of data blocks with timestamps; constructing a three-dimensional Bloom filter index matrix containing timestamps, data types and content features for the data blocks with the timestamps, associating a B + tree local index with an inverted global index through a hierarchical index structure, and establishing an index update priority queue by adopting a two-channel synchronization mechanism, performing real-time increment synchronization on the hotspot index through a heartbeat mechanism, performing batch synchronization on the cold data layer index according to a cold data synchronization period, predicting a data distribution probability through a distributed query statistics probability table during query, and initiating multi-path query in parallel based on probability weight, and the query path is dynamically optimized according to the node group storage medium type and the inter-node network transmission delay, so that rapid distribution adjustment and access of distributed storage are realized.

The invention discloses an intelligent traffic data storage method, system and device based on the Internet of Things and a medium, and relates to the technical field of electric digital data processing, and the method comprises the steps that an edge node packages received data and inserts the data into a local directed acyclic graph account book; dividing the verified data into Level-0 fragments according to a fixed duration window, and performing variable-length fragmentation based on a Rabin fingerprint algorithm; calculating Hash values of the variable-length fragments, and performing duplicate removal through a Bloom filter and a Hash index table in sequence; generating a local erasure block and a global erasure block for the deduplicated fragments, and storing complete copies of the key fragments on a plurality of storage nodes; and periodically checking a local erasure block and a complete copy, and triggering self-healing reconstruction when detecting that the fragment is lost. According to the method, the technical problems of high delay of distributed write-in consistency, high storage redundancy, complex cross-node time sequence correction, slow data recovery, poor system expansibility and the like are solved.

The invention discloses a code fingerprint-based open source component identification and vulnerability detection method, which comprises the following steps of: receiving a local or remote code, extracting a difference file, generating an AST and constructing a code attribute graph; sHA-256 Hash fingerprints and GNN semantic fingerprints are calculated for the function level sub-graphs to form mixed fingerprints, accurate matching is conducted through a Bloom filter, semantic matching is completed through nearest neighbor, and a component version is determined through a distribution difference algorithm. The method comprises the steps that firstly, a component identifier is mapped into a PURL or an SWID, an OSV / NVD library is inquired to obtain a CVE, comprehensive risks are calculated in combination with CVSS, EPSS and dependency depth, and an SBOM and a vulnerability report conforming to CycloneDX or SPDX are output. The method is high in speed and high in accuracy, and the open source risk can be automatically treated in continuous integration.

Systems, devices, and methods are provided for incremental partition updates. In at least one embodiment, indexing Bloom filters and incremental update Bloom filters are utilized to identify a set of candidate partitions of a distributed data set, such as in the context of a data lake. Queries may be processed using these candidate partitions.

The invention relates to a time series data acquisition method and device based on a dynamic time window and fingerprint deduplication. The method comprises the following steps: establishing a sliding time window with a variable length by taking the current system time as a reference, and only querying incremental time sequence data in the window; adaptively zooming the window by calculating the ratio of the collected data volume to the expected volume in real time; setting the trigger interval as half of the length of the sliding time window to form a time overlap, thereby capturing out-of-order data; key features of each piece of time series data are extracted, and data fingerprints are generated; a bloom filter is used for primary screening, and accurate duplicate removal is carried out in a key-value storage system; compared with an existing fixed polling scheme, the method has the advantages that resources can be dynamically adjusted along with data flow rate, system load is reduced, real-time performance is improved, data integrity is guaranteed by overlapped windows, zero repeated acquisition is guaranteed by fingerprint two-stage duplicate removal, and the method is applicable to efficient incremental synchronization scenes of transactional log-free time sequence databases such as InfluxDB and the like.

Methods, systems, and non-transitory computer readable storage media are disclosed for optimizing computer memory usage for lookup lists in computer memory via probabilistic data structures. For example, the disclosed system generates a probabilistic data structure (e.g., a Bloom filter) to represent data in a lookup list including multi-token items by hashing items of the lookup list to sets of bit values in a bit vector. The disclosed system classifies text content in a digital document by utilizing a maximum number of tokens from multi-token items in the lookup list to select and compare sets of sequential tokens in the digital document to the probabilistic data structure. The disclosed system also iteratively reduces the number of tokens in sets of sequential tokens for subsequent comparisons. Furthermore, in some aspects, the disclosed system causes a computing device to modify a digital document and / or database operations based on the classifications.

The invention relates to the technical field of information retrieval, and provides a privacy information retrieval system and method based on a block chain and function secret sharing, and the method comprises the steps: carrying out the encryption and keyword extraction of a data document; generating a Bloom filter index table between the keyword and the document based on the keyword trap door; uploading the encrypted file and the Bloom filter index table to a DSSP end, and transmitting the hash value of the encrypted file and the Bloom filter index table to a block chain for on-chain storage; storing the encrypted file in a distributed storage node under the chain according to the divided share; and the block chain calculates data of each column of the Bloom filter index table through consensus to obtain a message verification code, and the message verification code is stored on the block chain. According to the method, the index is constructed through symmetric encryption and the Bloom filter, and block chain evidence storage and FSS distributed storage are combined, so that data privacy protection and index credibility are realized. Through verification and tampering prevention, the problem that the existing FSS lacks verification and auditing mechanisms is solved.

The invention discloses a synchronization method and device for storage metadata, electronic equipment and a storage medium, and relates to the technical field of computers, a consistent Hash algorithm is adopted to realize mapping of data fragments and storage nodes, a monitoring node maintains the relation, a data change event is directionally sent to a corresponding partition, and the data change event is stored in the corresponding partition. Meanwhile, the Bloom filter is used for pre-checking events to filter invalid information, only local related changes are processed, the check value is transmitted, and the memory area is pre-registered, so that the data interaction mode between the nodes is optimized, and the defects of a full-connection mode are avoided; the technical problems that in an existing block storage metadata synchronization mechanism, due to the fact that a full connection mode is not optimized for node scale extension, the network connection number is increased in an exponential level mode, the system overhead is increased, and the cluster expansion capacity is limited can be solved, and the purposes of reducing the system overhead, improving the cluster expansion capacity and improving the cluster expansion efficiency are achieved. And the performance of the distributed block storage system is optimized.

The invention provides a privacy record linking method for low-quality data, which can be applied to the technical field of data fusion entity alignment. The method comprises the following steps: a link participant carries out multiple rounds of negotiation communication based on a link scene, and determines parameter information for subsequently executing a privacy record link; bloom filter coding operation is executed on respective data of link participants, the data missing number is recorded synchronously, and a coding table and a missing table are generated; the link unit analyzes the data of the coding table and the missing table by using the dynamic locality sensitive hash parameter, generates a record partitioning flag bit, and completes record partitioning according to the flag bit to form a partitioning group; a link unit extracts records in the corresponding blocks from the block group, marks the records in the blocks to form record pairs, and calculates a dynamic threshold value and Jaccard similarity of each record pair according to a reference filtering threshold value and a threshold value reduction amount; and in response to the Jaccard similarity being greater than or equal to the dynamic threshold, outputting the corresponding record pair identifier as a result.

The invention discloses a cross-domain authentication bridging method and system oriented to industrial Internet of Things interconnection, and belongs to the technical field of Internet of Things security, and the method comprises the steps of system initialization, cross-domain bridging, certificate verification, key exchange and secure communication, dynamic certificate updating and management and the like. According to the method, technical means such as ChaCha20, Poly1305, BLAKE2s, Ed25519, an elliptic curve Diffie-Hellman and the like are comprehensively utilized, a Bloom filter and a Merkle tree are utilized to realize incremental updating and rapid verification of a revoked list, and dynamic key generation and seamless switching are completed through domain division management. Therefore, the system has the characteristics of low time delay, high safety and fault tolerance, is suitable for large-scale industrial scenes, can remarkably reduce the bandwidth and calculation overhead, improves the equipment interconnection efficiency, enhances the safety, and is suitable for industrial field multi-domain collaboration.

The invention relates to a stable database query processing method and device based on predicate transfer, and the method comprises the steps: obtaining a target query statement; analyzing the target query statement to generate a target undirected graph; wherein the target undirected graph takes a base table corresponding to the target query statement as a node, and takes equality connection corresponding to the target query statement as an edge; converting the target undirected graph into a target directed connection graph; adopting a Bloom filter to pre-filter nodes in the target directed acyclic connection graph; and performing data query based on the pre-filtered target directed acyclic connection graph to obtain a query result. Through the method and the device, the redundant data in all the base tables in the target query statement is pre-filtered, and the method can be practically applied.

The invention discloses an RDMA Chain-based LSM Tree key value type database access method. The method comprises the following steps: a database client sends a request containing a version number, a key and an operation type to a database server main node; the server main node analyzes the request and processes the request according to the operation type; during write operation, a new serial number is generated, and data is inserted into a skip list of a memory table MemTable; when reading operation is carried out and the version number is smaller than 0, whether a key exists or not is checked through a Bloom filter, if yes, query is carried out at a secondary node through the RDMA technology, and if not, an SSTable file in a disk is directly accessed; and when the reading operation is performed and the version number is greater than or equal to 0, preferentially searching the data of the corresponding version in the MemTable, and if the data of the corresponding version fails, querying the SSTable. Through the RDMA technology, CPU occupation and network round-trip time are reduced, and the response speed and concurrent processing capacity of the system are improved.

The systems and methods disclosed herein orchestrate task execution among autonomous (or semi-autonomous) AI agentic models (“agents”) responsive to a received query by using a hierarchical semantic fingerprinting framework to generate semantic-aware fingerprints for the query and agents. Queries and descriptions of agent capabilities are processed by a series of hierarchical levels using locality-sensitive hash (LSH) functions, where subsequent layers encode more complex semantic information and generate longer hash values. The hash values are aggregated into a semantic fingerprint. A bloom filter cascade uses a series of increasingly accurate hierarchical bloom filters to reject agent fingerprints that differ from the query fingerprint. The remaining agent fingerprints are compared bitwise to the query fingerprint, and those closest to the query fingerprint are selected to generate a routing path for the query. Responses from the selected agents are aggregated into an output that is responsive to the input.

The invention discloses a single sign-on method and system based on flow interception and credential injection, and the method comprises the steps: a user completes UKey authentication through a gateway client, a gateway generates a session bound with the client and stores the session in a Redis cluster, and a dynamic TTL mechanism renews the session according to the operation of the user; when a user accesses an application system, a gateway captures traffic through a kernel layer, and a quintuple rule and a Bloom filter are adopted to intercept malicious requests; for a logged-in user, after the gateway queries the session state, an encrypted user certificate is called and a signature is added, and a request header is injected by a zero-copy technology and then forwarded to an application server. The system comprises a gateway client, an authentication gateway, a database and an application server. According to the invention, through a cryptographic algorithm, hardware-level key management and kernel layer flow optimization, the problems of complex integration, low security and performance bottleneck of a traditional single sign-on system are solved, and zero transformation access, microsecond-level delay and financial-level security compliance are realized.

A data lake for enterprise security is created from an asynchronous stream of security events by deduplicating objects and creating metadata related to downstream security functions. Deduplication of objects may be efficiently performed with a bloom filter as objects are ingested into the data lake. The objects may also be augmented with metadata arranged in schemas to facilitate monitoring and use within the data lake.

The invention discloses a digital fingerprint generation method for terminal equipment, and relates to the technical field of digital fingerprint generation, comprising the following steps: S1, collecting multi-dimensional hierarchical features; s2, privacy enhancement; s3, fingerprint anti-regeneration generation is carried out; and S4, fingerprint life cycle management. According to the terminal equipment digital fingerprint generation method, multi-dimensional fine-grained feature acquisition is realized from a non-sensitive layer, a semi-sensitive layer and a high-sensitive layer, a feature weight is optimized through a federated learning framework, fingerprint drift caused by hardware replacement and software updating is effectively resisted, and the user experience is improved. Dynamic Gaussian noise is injected into numerical semi-sensitive features to balance privacy protection and feature effectiveness, 2048-bit key Paillier homomorphic encryption is adopted for high-sensitive features, meanwhile, cross-scene tracking risks are blocked through scenarized sub-fingerprint generation and an original feature immediate destruction mechanism, and the privacy protection and feature effectiveness are improved. And an anti-duplication architecture fusing a Merkle tree and a bloom filter is introduced to greatly reduce the fingerprint coincidence collision rate.

The invention provides a data decoupling centralized query method and system based on MQ message triggering, and relates to the technical field of data query, and the method comprises the steps of receiving a message queue data change message, constructing a field, dividing check nodes by depending on a directed acyclic graph, carrying out data consistency verification, and pushing a transaction identifier to carry out concurrent operation scheduling. And collecting hotspot information by adopting a Bloom filter to adjust data fragmentation and cache distribution, generating a distributed query routing table, and distributing query requests according to the routing table. According to the method, the data consistency is improved, and the query performance is optimized.

The invention discloses an Internet of Things puncturable attribute-based encryption method based on a block chain and a Bloom filter, and is used for solving the problems of low dynamic permission revocation efficiency and difficult operation auditing in an Internet of Things environment. The method realizes a core function through the following steps: an authorization mechanism initializes system parameters and segments a master key; a data owner defines an access strategy and initializes the Bloom filter during encryption; when the permission is revoked, a puncture tuple containing a label binding component, a strategy binding component and a cross validation component is generated, and the ciphertext is updated after verification of a block chain smart contract; and during decryption, the cloud server firstly screens the validity of the user tag through a Bloom filter, and then decrypts the data in combination with an attribute strategy. The method has the advantages that the complexity of puncture verification is reduced by the Bloom filter, and the resource consumption is remarkably reduced; two-factor binding and cross validation ensure that puncture operation cannot be forged; and the block chain completely records key generation, puncture and decryption operations, so that operation transparency and auditing performance are realized.

The invention provides a high-speed mode matching device and method based on an FPGA, and the device comprises an extraction module which is used for carrying out the multi-dimensional feature extraction of a rule set; the clustering module is used for generating rule categories; the Bloom filtering module is used for screening the input data; the Hash matching module is used for verifying whether the data flow is matched with a known mode or not; the probability verification module is used for calculating and verifying a hash matching passing probability; the regular matching module is used for matching complex modes; the feedback module is used for dynamically adjusting Bloom filter parameters, hash table distribution and probability formula coefficients; and the matching result reporting module is used for reporting the matching information. According to the method, the balance of high efficiency, accuracy and expandability can be realized in a multi-mode matching task, the resource utilization efficiency of an FPGA (Field Programmable Gate Array) is improved, and the matching requirement when a mode matching rule set is relatively large is met.

A method and system for detecting a denial of service (DOS) attack when packets used in the DOS attack are user datagram protocol (UDP) packets, are disclosed. The method includes obtaining values from an application header of a UDP packet; and when, based on use of at least one modified probabilistic Bloom filter (PMBF) for a destination to which the UDP packet indicates that it is destined, the obtained values are not values expected to be found in a UDP packet destined for the destination: increasing an uncommon per sample (ups) estimate for the PMBF; and when the increased uncommon per sample estimate is greater than an upper ups threshold: setting an alarm state to on; suspending update of the PMBF counters; and initiating a mitigation action at least with respect to the UDP packet.

The invention discloses a communication efficient data opening and sharing method and system based on privacy set intersection. In order to solve the problem that when the number of participants is large, the communication bandwidth of the participants in a privacy calculation scene is limited, the invention proposes that different element insertion sequences have influences on the calculation result of the confusion Bloom filter for the first time; an efficient optimal element insertion sequence search algorithm assisted by a counting bloom filter is provided based on a greedy strategy, and meanwhile, a theoretical lower bound and a theoretical upper bound for improving the storage space of the confusion bloom filter under a new algorithm are analyzed; a secret sharing principle of an improved confusion bloom filter is changed from XOR calculation to additive calculation, so that a related algorithm of the improved confusion bloom filter can perform homomorphic calculation in a ciphertext encrypted by a public key; based on the improved confusion Bloom filter, a multi-party privacy set intersection protocol with high communication efficiency is realized, and the protocol can be used for calculating a multi-party intersection and also can be used for calculating a threshold multi-party intersection of a self-defined threshold.

The invention discloses an intersection element security calculation method and system oriented to data opening and sharing, and the method comprises the steps: enabling a participant to generate a Bloom filter through employing a privacy data set when the participant operates the intersection element security calculation for the first time, and generating a public key encrypted Bloom filter through employing a public key of a threshold public key encryption system meeting the homomorphism of addition. Secondly, based on the similarity between the Bloom filter and the counting Bloom filter, when the data set of the participant is updated, the counting Bloom filter of the local cache can be efficiently updated firstly, then the Bloom filter encrypted by the public key is updated by using a more efficient homomorphic addition, and the participant not updating the data set does not need to re-encrypt the data set any more. The multi-party security comparison protocol only needs to be operated in cooperation with the participant playing the server, so that multiplexing of the data set encrypted by the public key is realized on the premise of not influencing the protocol security. The system comprises a parameter confirmation module, a random selection module, a public key encryption module, a judgment module, a client updating module and a server updating module.

The invention discloses an intelligent operation and maintenance management and control method and system based on a digital twinning wind power plant, and relates to the technical field of digital twinning, and the method comprises the steps: obtaining a wind power plant fan time sequence signal, constructing a feature vector, carrying out the Hash mapping, and storing the feature vector into a segmented Hash table; judging an operation mode through a Bloom filter, generating a label triple stream and summarizing abnormal hash keys; counting the Hash value frequency in the time window and marking a high-risk value, and generating a Hash frequency distribution vector; calculating mutual information between nodes, and constructing an adjacent forest through a minimum generation forest; carrying out sampling and low-frequency signal inflection point storage on the sample flow; constructing an exception decision-making tree, and judging exceptional nodes; extracting weighted sub-graphs and dividing the weighted sub-graphs into fault mode clusters; and calculating a fault urgency degree score and generating an operation and maintenance work order. Through a multi-source time sequence signal, Hash mapping, a decision tree and a spectrum division algorithm, wind power plant fault detection, mode recognition and operation and maintenance scheduling are optimized, and the operation efficiency and sustainability are improved.

The invention discloses a large-batch data processing method and system, and relates to the technical field of large-batch data processing, and the method comprises the steps: carrying out the real-time collection of multi-source heterogeneous data through employing a distributed message queue, and obtaining an original data flow; performing fragmentation processing on the original data stream by adopting an entropy weight dynamic fragmentation algorithm to obtain uniformly distributed data fragments; constructing a hybrid model combining HyperLogLog + + and a Bloom filter based on the original data stream, and de-duplicating the data fragments to obtain a unique data set; performing iterative calculation on the unique data set by adopting a delta-convergence criterion to obtain a final analysis result; the analysis result is written into a distributed file platform through column storage; multi-source heterogeneous data are collected in real time by adopting a distributed message queue, an efficient data collection and transmission mechanism is realized, the method can easily deal with inflow of large-scale data through the design, and the expandability and stability of the method are enhanced.

The invention provides an intelligent accelerated data reading method and system for a solid state disk, and the method comprises the steps: reading basic information of each data block in the solid state disk when a data reading request is received, and constructing an information fingerprint of each data block based on the basic information; constructing an information fingerprint of request data based on the data reading request, and eliminating unmatched data blocks by adopting a Bloom filter; performing similarity calculation on the information fingerprints of the request data and the information fingerprints of the excluded data blocks by using a locality sensitive hash algorithm, and positioning the data block most similar to the request data; and reading the request data from the solid state disk according to a positioning result. According to the method and the device, unmatched data blocks are eliminated by adopting a Bloom filter; similarity calculation is carried out on the information fingerprints of the request data and the information fingerprints of the excluded data blocks, the data block most similar to the request data is positioned, and the defects of long time consumption and system resource waste during current data reading are overcome.

The embodiment of the invention provides a batch retrieval method based on a verifiable bloom filter and electronic equipment, and relates to the technical field of data security. The method comprises the following steps: converting a server data set into a key value pair form, and mapping keywords in the data set in the key value pair form into equal repeated codes through an equal repeated code mapping algorithm to obtain an equal repeated code data set; determining a storage position of a second verifiable bloom filter according to a keyword in the equal-repetition-code data set, and storing a data entry corresponding to the keyword in the equal-repetition-code data set and a check code generated based on the data entry to the storage position; the client receives the query keyword set and maps the query keyword set to the first verifiable Bloom filter; and taking each bit of the first verifiable Bloom filter as a selection vector to execute an oblivious transmission protocol to obtain a return result of the server side. The embodiment provided by the invention provides a safe and efficient batch retrieval method based on the verifiable Bloom filter.

The invention discloses an integrity checking method in a file transmission process and related equipment, which are applied to a data sending end and comprise the following steps: acquiring a to-be-sent file set; for each file in the to-be-sent file set, calculating an integrity label of the file by adopting a verification mode corresponding to the file; the verification mode is related to the data volume of the file; the form of the integrity label is a hash value form; processing the integrity label corresponding to each file by applying a set bloom filter to obtain a target integrity label; the form of the target integrity label is a hash value form; and sending the to-be-sent file set and the target integrity label to a data receiving end, so that the data receiving end verifies whether each file in the received to-be-sent file set is complete or not according to the target integrity label. According to the embodiment of the invention, the data transmission quantity is small, the consumption of hardware resources is reduced, and the data transmission speed is high, so that the speed of a data verification process is accelerated, and the confidentiality is enhanced.

The invention provides a data duplicate removal method, electronic equipment and a storage medium. According to the method, smooth data transition is achieved through a double-bloom filter alternating mechanism. In first time period processing, query and addition operations are only executed on a first bloom filter of a current time window. And second time period initialization: creating a new second bloom filter at the first unit time of the second time period as a next window preparation filter. According to the double-writing preheating mechanism, data are written into the first Bloom filter and the second Bloom filter at the same time in the second time period, and the missed judgment problem during window switching is avoided. According to window switching logic, old filter resources are released, the preliminary filter is marked as a new main filter, and monthly / periodic seamless connection is achieved. According to the method, the misjudgment rate is controlled through a double-bloom filter time window switching mechanism, the data duplicate removal accuracy and the system resource utilization rate can be improved, and the method is suitable for the efficient duplicate removal requirement in a mass data collection scene.

The invention provides an enhanced symmetric searchable encryption method and system, and relates to the field of data security and privacy protection. The method comprises the steps that a plaintext document set is symmetrically encrypted, keywords are extracted, and an encryption index token is generated through a pseudo-random function; constructing a reverse index structure, and mapping the token to a document identifier; dynamic insertion, deletion and modification of documents are supported, and indexes are synchronously updated in real time; cipher query is realized by generating a search token, and after the server returns a matching result, the client decrypts the matching result to obtain an original text. According to the system, through multi-keyword Boolean logic query, an index structure is compressed by adopting a Bloom filter, the retrieval efficiency is improved in combination with a jump pointer or a B + tree, a false query strategy is introduced to hide an access mode, and the anti-attack ability is enhanced. Compared with an existing scheme, the method has the advantages that the searchability, the system performance and the privacy protection capability of the encrypted data are improved, and the method is suitable for security data retrieval application in cloud computing, database systems and block chain environments.

An authentication information storage method, system, device and product of an edge computing device relate to the technical field of edge computing, and the method comprises the following steps: receiving first revoked authentication information in response to a service instance of a cluster, according to the effective deadline of the first revocation authentication information and a preset time interval corresponding to a plurality of fragmented Bloom filters of the service instance, determining a first fragmented Bloom filter, and storing the first revocation authentication information into the first fragmented Bloom filter; and in response to the expiration of all the revoked authentication information stored in the first fragmentation Bloom filter, emptying all the revoked authentication information stored in the first fragmentation Bloom filter. According to the technical scheme, the revoked authentication information is stored in the fragmented mode according to the effective deadline, expired deletion of the revoked authentication information in the fragmented Bloom filter is achieved, a guarantee is provided for local authentication revoked inspection of the big data magnitude in the edge computing scene, the storage space is saved, and the accuracy and reliability of authentication revoked inspection can be guaranteed.