461 results about "Bloom filter" patented technology

The invention relates to the technical field of information retrieval, and discloses a distributed storage and indexing method, which comprises the following steps of: dynamically fragmenting an original file through an improved consistent Hash algorithm to generate a plurality of data blocks with timestamps; constructing a three-dimensional Bloom filter index matrix containing timestamps, data types and content features for the data blocks with the timestamps, associating a B + tree local index with an inverted global index through a hierarchical index structure, and establishing an index update priority queue by adopting a two-channel synchronization mechanism, performing real-time increment synchronization on the hotspot index through a heartbeat mechanism, performing batch synchronization on the cold data layer index according to a cold data synchronization period, predicting a data distribution probability through a distributed query statistics probability table during query, and initiating multi-path query in parallel based on probability weight, and the query path is dynamically optimized according to the node group storage medium type and the inter-node network transmission delay, so that rapid distribution adjustment and access of distributed storage are realized.

The invention discloses a P4 switch processor core-oriented multi-service traffic packet scheduling method, which mainly comprises the following steps of: proposing a feature extraction method based on a statistical interval and Sketch, integrating data packet time correlation and traffic dynamic change, optimizing a Bloom filter conflict detection mechanism, and improving the adaptability and accuracy of an algorithm in a limited resource environment; a decision tree model is combined with scheduling, a decision tree is reconstructed to adapt to an operation mode and a data processing requirement of a P4 switch, and high-precision service flow classification under linear speed operation is ensured; a packet scheduling method AD-PIFO is designed based on a variable priority, the forwarding priority of a data packet is dynamically adjusted, the QoS requirements of different services are met, and the overall performance and efficiency of the switch are improved. The method is outstanding in high-priority traffic throughput and low-delay performance, is superior to similar algorithms in medium-low priority traffic throughput, is stable and effective under different load conditions, is reasonable in extra time overhead of data packet transmission, and is widely applied to SDN network optimization data packet scheduling scenes.

The invention discloses an intelligent traffic data storage method, system and device based on the Internet of Things and a medium, and relates to the technical field of electric digital data processing, and the method comprises the steps that an edge node packages received data and inserts the data into a local directed acyclic graph account book; dividing the verified data into Level-0 fragments according to a fixed duration window, and performing variable-length fragmentation based on a Rabin fingerprint algorithm; calculating Hash values of the variable-length fragments, and performing duplicate removal through a Bloom filter and a Hash index table in sequence; generating a local erasure block and a global erasure block for the deduplicated fragments, and storing complete copies of the key fragments on a plurality of storage nodes; and periodically checking a local erasure block and a complete copy, and triggering self-healing reconstruction when detecting that the fragment is lost. According to the method, the technical problems of high delay of distributed write-in consistency, high storage redundancy, complex cross-node time sequence correction, slow data recovery, poor system expansibility and the like are solved.

The invention discloses a code fingerprint-based open source component identification and vulnerability detection method, which comprises the following steps of: receiving a local or remote code, extracting a difference file, generating an AST and constructing a code attribute graph; sHA-256 Hash fingerprints and GNN semantic fingerprints are calculated for the function level sub-graphs to form mixed fingerprints, accurate matching is conducted through a Bloom filter, semantic matching is completed through nearest neighbor, and a component version is determined through a distribution difference algorithm. The method comprises the steps that firstly, a component identifier is mapped into a PURL or an SWID, an OSV / NVD library is inquired to obtain a CVE, comprehensive risks are calculated in combination with CVSS, EPSS and dependency depth, and an SBOM and a vulnerability report conforming to CycloneDX or SPDX are output. The method is high in speed and high in accuracy, and the open source risk can be automatically treated in continuous integration.

Systems, devices, and methods are provided for incremental partition updates. In at least one embodiment, indexing Bloom filters and incremental update Bloom filters are utilized to identify a set of candidate partitions of a distributed data set, such as in the context of a data lake. Queries may be processed using these candidate partitions.

The invention relates to a time series data acquisition method and device based on a dynamic time window and fingerprint deduplication. The method comprises the following steps: establishing a sliding time window with a variable length by taking the current system time as a reference, and only querying incremental time sequence data in the window; adaptively zooming the window by calculating the ratio of the collected data volume to the expected volume in real time; setting the trigger interval as half of the length of the sliding time window to form a time overlap, thereby capturing out-of-order data; key features of each piece of time series data are extracted, and data fingerprints are generated; a bloom filter is used for primary screening, and accurate duplicate removal is carried out in a key-value storage system; compared with an existing fixed polling scheme, the method has the advantages that resources can be dynamically adjusted along with data flow rate, system load is reduced, real-time performance is improved, data integrity is guaranteed by overlapped windows, zero repeated acquisition is guaranteed by fingerprint two-stage duplicate removal, and the method is applicable to efficient incremental synchronization scenes of transactional log-free time sequence databases such as InfluxDB and the like.

Systems and methods are described for batch materialization of an incremental change data capture (CDC) changeset with full row changes. The primary keys are extracted from the incremental CDC changeset and an indication of the extracted primary keys are broadcast to a plurality of executors. The primary keys may be added to Bloom filter or a plurality of Bloom filters that are broadcast to the executors. Each executor filters a baseline data table based on the extracted primary keys to generate a baseline match dataframe with all primary keys matching the extracted primary keys, and a baseline unmatched dataframe with all primary keys not matching the extracted primary keys. Each executor receives full row changes from a partitioned incremental CDC changeset and combines the changes with the baseline unmatched dataframe to produce a final changed baseline data table.

Methods, systems, and non-transitory computer readable storage media are disclosed for optimizing computer memory usage for lookup lists in computer memory via probabilistic data structures. For example, the disclosed system generates a probabilistic data structure (e.g., a Bloom filter) to represent data in a lookup list including multi-token items by hashing items of the lookup list to sets of bit values in a bit vector. The disclosed system classifies text content in a digital document by utilizing a maximum number of tokens from multi-token items in the lookup list to select and compare sets of sequential tokens in the digital document to the probabilistic data structure. The disclosed system also iteratively reduces the number of tokens in sets of sequential tokens for subsequent comparisons. Furthermore, in some aspects, the disclosed system causes a computing device to modify a digital document and / or database operations based on the classifications.

The invention provides a monitoring data linkage method between code-free applications, automatic identification and mapping of data structures between the applications are realized by constructing a metadata description model and a field similarity function, and data change detection is performed by adopting an improved locality sensitive hash algorithm. Through a distributed transaction control mechanism, data consistency in cross-application data linkage is ensured, and a multi-stage bloom filter and a distributed transaction feature evaluation model are designed. Furthermore, data synchronization is realized by utilizing a synchronous vector field, and data synchronization batch processing and concurrency adjustment are optimized in combination with a Gaussian divergence theorem. Besides, the invention provides abnormal recovery path optimization based on the minimum action quantity, the reliability of data linkage is ensured, and a dynamic data security protection mechanism is constructed through a wave function, a diffusion equation and nonlinear mapping.

The invention relates to the technical field of information retrieval, and provides a privacy information retrieval system and method based on a block chain and function secret sharing, and the method comprises the steps: carrying out the encryption and keyword extraction of a data document; generating a Bloom filter index table between the keyword and the document based on the keyword trap door; uploading the encrypted file and the Bloom filter index table to a DSSP end, and transmitting the hash value of the encrypted file and the Bloom filter index table to a block chain for on-chain storage; storing the encrypted file in a distributed storage node under the chain according to the divided share; and the block chain calculates data of each column of the Bloom filter index table through consensus to obtain a message verification code, and the message verification code is stored on the block chain. According to the method, the index is constructed through symmetric encryption and the Bloom filter, and block chain evidence storage and FSS distributed storage are combined, so that data privacy protection and index credibility are realized. Through verification and tampering prevention, the problem that the existing FSS lacks verification and auditing mechanisms is solved.

The invention belongs to the spaceflight communication technology, and particularly relates to a fast routing lookup method applied to a spaceborne router, which comprises the following steps: constructing a heterogeneous network processing platform with a control surface and a data surface separated; generating a routing forwarding table on a control plane, and processing the routing forwarding table by adopting a leaf-push type conversion method to obtain an updated routing forwarding table; constructing a perfect hash function by adopting a ZPHF algorithm to obtain a hash mapping table; carrying out Bloom mapping by adopting a perfect hash function to obtain a Bloom filter mapping table; injecting the Hash mapping table and the Bloom filter mapping table into a data surface; when a node initiates a routing lookup request, extracting a destination IP address, querying a bit in the Bloom filter mapping table, if the bit is 1, accessing the Hash mapping table for matching, if the matching is successful, completing routing lookup, and if the matching is unsuccessful, updating the destination IP address and performing lookup again; according to the method, the considerable search speed can be guaranteed under the condition of relatively low storage consumption.

The invention discloses a synchronization method and device for storage metadata, electronic equipment and a storage medium, and relates to the technical field of computers, a consistent Hash algorithm is adopted to realize mapping of data fragments and storage nodes, a monitoring node maintains the relation, a data change event is directionally sent to a corresponding partition, and the data change event is stored in the corresponding partition. Meanwhile, the Bloom filter is used for pre-checking events to filter invalid information, only local related changes are processed, the check value is transmitted, and the memory area is pre-registered, so that the data interaction mode between the nodes is optimized, and the defects of a full-connection mode are avoided; the technical problems that in an existing block storage metadata synchronization mechanism, due to the fact that a full connection mode is not optimized for node scale extension, the network connection number is increased in an exponential level mode, the system overhead is increased, and the cluster expansion capacity is limited can be solved, and the purposes of reducing the system overhead, improving the cluster expansion capacity and improving the cluster expansion efficiency are achieved. And the performance of the distributed block storage system is optimized.

The invention provides a sensitive data protection system and method based on pseudonymous encryption, and relates to the field of sensitive data protection, and the system comprises the following modules: a dynamic pseudonym generation module which is used for receiving original sensitive data in real time, and generating a dynamic pseudonym based on a quantum random number seed and a timeliness noise factor; and the regulation self-adaptive decision module is connected with the dynamic pseudonym generation module. According to the method, a quantum random number seed # imgabs0 #, a timeliness noise factor # imgabs1 # and a life cycle lt are adopted; the dynamic pseudonym is unpredictable and is short-term effective, the replay attack risk is remarkably reduced, key synthesis and data reduction are realized through an SGX secure enclave in a GDPR mode, the execution success rate of the forgotten right is ensured, and differential privacy k = 3, delta = 0.1 and Bloom Filter are adopted for desensitization in a CCPA mode, so that the reidentification risk is reduced.

The invention provides a privacy record linking method for low-quality data, which can be applied to the technical field of data fusion entity alignment. The method comprises the following steps: a link participant carries out multiple rounds of negotiation communication based on a link scene, and determines parameter information for subsequently executing a privacy record link; bloom filter coding operation is executed on respective data of link participants, the data missing number is recorded synchronously, and a coding table and a missing table are generated; the link unit analyzes the data of the coding table and the missing table by using the dynamic locality sensitive hash parameter, generates a record partitioning flag bit, and completes record partitioning according to the flag bit to form a partitioning group; a link unit extracts records in the corresponding blocks from the block group, marks the records in the blocks to form record pairs, and calculates a dynamic threshold value and Jaccard similarity of each record pair according to a reference filtering threshold value and a threshold value reduction amount; and in response to the Jaccard similarity being greater than or equal to the dynamic threshold, outputting the corresponding record pair identifier as a result.

The invention discloses a cross-domain authentication bridging method and system oriented to industrial Internet of Things interconnection, and belongs to the technical field of Internet of Things security, and the method comprises the steps of system initialization, cross-domain bridging, certificate verification, key exchange and secure communication, dynamic certificate updating and management and the like. According to the method, technical means such as ChaCha20, Poly1305, BLAKE2s, Ed25519, an elliptic curve Diffie-Hellman and the like are comprehensively utilized, a Bloom filter and a Merkle tree are utilized to realize incremental updating and rapid verification of a revoked list, and dynamic key generation and seamless switching are completed through domain division management. Therefore, the system has the characteristics of low time delay, high safety and fault tolerance, is suitable for large-scale industrial scenes, can remarkably reduce the bandwidth and calculation overhead, improves the equipment interconnection efficiency, enhances the safety, and is suitable for industrial field multi-domain collaboration.

The invention relates to a stable database query processing method and device based on predicate transfer, and the method comprises the steps: obtaining a target query statement; analyzing the target query statement to generate a target undirected graph; wherein the target undirected graph takes a base table corresponding to the target query statement as a node, and takes equality connection corresponding to the target query statement as an edge; converting the target undirected graph into a target directed connection graph; adopting a Bloom filter to pre-filter nodes in the target directed acyclic connection graph; and performing data query based on the pre-filtered target directed acyclic connection graph to obtain a query result. Through the method and the device, the redundant data in all the base tables in the target query statement is pre-filtered, and the method can be practically applied.

The technology disclosed herein provides a mechanism to avoid double counting when generating metrics from monitoring events received from an endpoint (e.g., end user's web browser, or mobile application). A unique identifier is assigned to each monitoring event matching a metric definition. When the number of unique identifiers assigned is below a predetermined threshold, a deduplication system determines whether the unique identifier for a given monitoring event is duplicative of any other unique identifiers, corresponding to other monitoring events, using a hash set. When the number of unique identifiers exceeds the predetermined threshold, the unique identifiers are automatically added to a probabilistic data structure, such as a scalable Bloom filter. In this scenario, the deduplication system would determine whether the unique identifier for the given monitoring event is duplicative of any other identifiers in the probabilistic data structure.

The invention discloses an RDMA Chain-based LSM Tree key value type database access method. The method comprises the following steps: a database client sends a request containing a version number, a key and an operation type to a database server main node; the server main node analyzes the request and processes the request according to the operation type; during write operation, a new serial number is generated, and data is inserted into a skip list of a memory table MemTable; when reading operation is carried out and the version number is smaller than 0, whether a key exists or not is checked through a Bloom filter, if yes, query is carried out at a secondary node through the RDMA technology, and if not, an SSTable file in a disk is directly accessed; and when the reading operation is performed and the version number is greater than or equal to 0, preferentially searching the data of the corresponding version in the MemTable, and if the data of the corresponding version fails, querying the SSTable. Through the RDMA technology, CPU occupation and network round-trip time are reduced, and the response speed and concurrent processing capacity of the system are improved.

The systems and methods disclosed herein orchestrate task execution among autonomous (or semi-autonomous) AI agentic models (“agents”) responsive to a received query by using a hierarchical semantic fingerprinting framework to generate semantic-aware fingerprints for the query and agents. Queries and descriptions of agent capabilities are processed by a series of hierarchical levels using locality-sensitive hash (LSH) functions, where subsequent layers encode more complex semantic information and generate longer hash values. The hash values are aggregated into a semantic fingerprint. A bloom filter cascade uses a series of increasingly accurate hierarchical bloom filters to reject agent fingerprints that differ from the query fingerprint. The remaining agent fingerprints are compared bitwise to the query fingerprint, and those closest to the query fingerprint are selected to generate a routing path for the query. Responses from the selected agents are aggregated into an output that is responsive to the input.

The invention discloses a single sign-on method and system based on flow interception and credential injection, and the method comprises the steps: a user completes UKey authentication through a gateway client, a gateway generates a session bound with the client and stores the session in a Redis cluster, and a dynamic TTL mechanism renews the session according to the operation of the user; when a user accesses an application system, a gateway captures traffic through a kernel layer, and a quintuple rule and a Bloom filter are adopted to intercept malicious requests; for a logged-in user, after the gateway queries the session state, an encrypted user certificate is called and a signature is added, and a request header is injected by a zero-copy technology and then forwarded to an application server. The system comprises a gateway client, an authentication gateway, a database and an application server. According to the invention, through a cryptographic algorithm, hardware-level key management and kernel layer flow optimization, the problems of complex integration, low security and performance bottleneck of a traditional single sign-on system are solved, and zero transformation access, microsecond-level delay and financial-level security compliance are realized.

The invention relates to a key value caching system based on a partition namespace solid state disk. The system comprises an object and metadata insertion module, an object searching module and an object expelling module. The key value is mapped to the offset in the logic partition through a hash function, and efficient search is realized by using a page-level bloom filter; object write-in data and a first metadata buffer area are written into a disk in batches by adopting a log structure to form a first-in first-out logic partition chain; caching the hot object metadata by the system, and performing classified management on the object according to the access frequency: loading the object metadata of which the access frequency exceeds a first preset frequency after unloading, and dynamically loading the object metadata of which the access frequency is lower than a second preset frequency after unloading; in the eviction process, in combination with the second metadata buffer area and the real-time access condition, high-frequency access objects are selectively recycled. Through Hash mapping, the page-level Bloom filter and the object management based on the access frequency, the performance and efficiency of the key value caching system are improved.

An implementation is a method for operating a cache memory in a computing system, receiving a request for a first data item from the cache memory of the computing system, the first data item having an associated tag value. The method also includes performing a lookup in a bloom filter for the tag value associated with the first data item. The method also includes performing a lookup in the cache memory for the requested first data item based on the lookup in the bloom filter. The method also includes updating the bloom filter based on results of the lookup in the cache memory for the requested first data item.

A data lake for enterprise security is created from an asynchronous stream of security events by deduplicating objects and creating metadata related to downstream security functions. Deduplication of objects may be efficiently performed with a bloom filter as objects are ingested into the data lake. The objects may also be augmented with metadata arranged in schemas to facilitate monitoring and use within the data lake.

The invention discloses a digital fingerprint generation method for terminal equipment, and relates to the technical field of digital fingerprint generation, comprising the following steps: S1, collecting multi-dimensional hierarchical features; s2, privacy enhancement; s3, fingerprint anti-regeneration generation is carried out; and S4, fingerprint life cycle management. According to the terminal equipment digital fingerprint generation method, multi-dimensional fine-grained feature acquisition is realized from a non-sensitive layer, a semi-sensitive layer and a high-sensitive layer, a feature weight is optimized through a federated learning framework, fingerprint drift caused by hardware replacement and software updating is effectively resisted, and the user experience is improved. Dynamic Gaussian noise is injected into numerical semi-sensitive features to balance privacy protection and feature effectiveness, 2048-bit key Paillier homomorphic encryption is adopted for high-sensitive features, meanwhile, cross-scene tracking risks are blocked through scenarized sub-fingerprint generation and an original feature immediate destruction mechanism, and the privacy protection and feature effectiveness are improved. And an anti-duplication architecture fusing a Merkle tree and a bloom filter is introduced to greatly reduce the fingerprint coincidence collision rate.

The invention provides a data decoupling centralized query method and system based on MQ message triggering, and relates to the technical field of data query, and the method comprises the steps of receiving a message queue data change message, constructing a field, dividing check nodes by depending on a directed acyclic graph, carrying out data consistency verification, and pushing a transaction identifier to carry out concurrent operation scheduling. And collecting hotspot information by adopting a Bloom filter to adjust data fragmentation and cache distribution, generating a distributed query routing table, and distributing query requests according to the routing table. According to the method, the data consistency is improved, and the query performance is optimized.

The invention discloses an Internet of Things puncturable attribute-based encryption method based on a block chain and a Bloom filter, and is used for solving the problems of low dynamic permission revocation efficiency and difficult operation auditing in an Internet of Things environment. The method realizes a core function through the following steps: an authorization mechanism initializes system parameters and segments a master key; a data owner defines an access strategy and initializes the Bloom filter during encryption; when the permission is revoked, a puncture tuple containing a label binding component, a strategy binding component and a cross validation component is generated, and the ciphertext is updated after verification of a block chain smart contract; and during decryption, the cloud server firstly screens the validity of the user tag through a Bloom filter, and then decrypts the data in combination with an attribute strategy. The method has the advantages that the complexity of puncture verification is reduced by the Bloom filter, and the resource consumption is remarkably reduced; two-factor binding and cross validation ensure that puncture operation cannot be forged; and the block chain completely records key generation, puncture and decryption operations, so that operation transparency and auditing performance are realized.

The invention discloses a service data real-time pushing method and device under a multi-tenant architecture, and the method comprises the steps: carrying out the filtering through matching a Bloom filter according to the data type of service data and a unique tenant identifier when monitoring that a service platform generates the service data, and obtaining to-be-pushed data; pushing configuration information is determined according to the data type of the to-be-pushed data and the unique tenant identifier; performing field conversion on the to-be-pushed data to generate field data; and encrypting the field data according to the push configuration information and pushing the field data to the subscription user side. Through table storage, cache data structure design and configuration, a monitoring updating strategy is changed, so that the throughput and pushing efficiency of data pushing are improved, meanwhile, the data pushing configuration updating can take effect in time, and the pushing flexibility of service data is improved.

The invention discloses a data consistency verification method and device, equipment and a medium. Original data and migration data in the source database and the target database are divided into unit original data and unit migration data through the same division rule; wherein the source database represents a data source of completed data migration, and the target database represents a data migration target of the data migration; calculating k first mapping values of each unit original data through a Bloom filter; based on each first mapping value, assigning 1 to a corresponding position of a binary array coordinate of the Bloom filter; and calculating k second mapping values of each unit migration data through the Bloom filter, determining whether the value of each second mapping value at the corresponding position of the binary array coordinate of the Bloom filter is 1, and if not, determining that the unit migration data is inconsistent data. According to the technical scheme, the verification efficiency of the data consistency after data migration can be improved.

The invention provides a high-speed mode matching device and method based on an FPGA, and the device comprises an extraction module which is used for carrying out the multi-dimensional feature extraction of a rule set; the clustering module is used for generating rule categories; the Bloom filtering module is used for screening the input data; the Hash matching module is used for verifying whether the data flow is matched with a known mode or not; the probability verification module is used for calculating and verifying a hash matching passing probability; the regular matching module is used for matching complex modes; the feedback module is used for dynamically adjusting Bloom filter parameters, hash table distribution and probability formula coefficients; and the matching result reporting module is used for reporting the matching information. According to the method, the balance of high efficiency, accuracy and expandability can be realized in a multi-mode matching task, the resource utilization efficiency of an FPGA (Field Programmable Gate Array) is improved, and the matching requirement when a mode matching rule set is relatively large is met.

The invention provides a cross-platform live broadcast visitor data fusion and routing load balancing method and system, and relates to the technical field of load balancing, and the method comprises the steps: filtering repeated visitor identifiers through a Bloom filter array, calculating the number of independent visitors through a logarithm radix counting algorithm, and dividing the priority of the visitors based on a data flow entropy value; constructing an annular routing structure by using a consistent Hash algorithm, mapping visitors to virtual nodes, and performing fragmentation routing based on priorities; the response time of the virtual nodes is recorded in real time, abnormal nodes are identified, associated visitors are migrated to standby nodes, and dynamic load balancing is achieved; the visitor data processing efficiency and the system stability of the live broadcast platform are effectively improved, the load pressure of the server is reduced, and the fluency of the live broadcast service is guaranteed.