261 results about "Bloom filter" patented technology

The invention relates to the technical field of information retrieval, and discloses a distributed storage and indexing method, which comprises the following steps of: dynamically fragmenting an original file through an improved consistent Hash algorithm to generate a plurality of data blocks with timestamps; constructing a three-dimensional Bloom filter index matrix containing timestamps, data types and content features for the data blocks with the timestamps, associating a B + tree local index with an inverted global index through a hierarchical index structure, and establishing an index update priority queue by adopting a two-channel synchronization mechanism, performing real-time increment synchronization on the hotspot index through a heartbeat mechanism, performing batch synchronization on the cold data layer index according to a cold data synchronization period, predicting a data distribution probability through a distributed query statistics probability table during query, and initiating multi-path query in parallel based on probability weight, and the query path is dynamically optimized according to the node group storage medium type and the inter-node network transmission delay, so that rapid distribution adjustment and access of distributed storage are realized.

The invention discloses a code fingerprint-based open source component identification and vulnerability detection method, which comprises the following steps of: receiving a local or remote code, extracting a difference file, generating an AST and constructing a code attribute graph; sHA-256 Hash fingerprints and GNN semantic fingerprints are calculated for the function level sub-graphs to form mixed fingerprints, accurate matching is conducted through a Bloom filter, semantic matching is completed through nearest neighbor, and a component version is determined through a distribution difference algorithm. The method comprises the steps that firstly, a component identifier is mapped into a PURL or an SWID, an OSV / NVD library is inquired to obtain a CVE, comprehensive risks are calculated in combination with CVSS, EPSS and dependency depth, and an SBOM and a vulnerability report conforming to CycloneDX or SPDX are output. The method is high in speed and high in accuracy, and the open source risk can be automatically treated in continuous integration.

The invention relates to a time series data acquisition method and device based on a dynamic time window and fingerprint deduplication. The method comprises the following steps: establishing a sliding time window with a variable length by taking the current system time as a reference, and only querying incremental time sequence data in the window; adaptively zooming the window by calculating the ratio of the collected data volume to the expected volume in real time; setting the trigger interval as half of the length of the sliding time window to form a time overlap, thereby capturing out-of-order data; key features of each piece of time series data are extracted, and data fingerprints are generated; a bloom filter is used for primary screening, and accurate duplicate removal is carried out in a key-value storage system; compared with an existing fixed polling scheme, the method has the advantages that resources can be dynamically adjusted along with data flow rate, system load is reduced, real-time performance is improved, data integrity is guaranteed by overlapped windows, zero repeated acquisition is guaranteed by fingerprint two-stage duplicate removal, and the method is applicable to efficient incremental synchronization scenes of transactional log-free time sequence databases such as InfluxDB and the like.

Methods, systems, and non-transitory computer readable storage media are disclosed for optimizing computer memory usage for lookup lists in computer memory via probabilistic data structures. For example, the disclosed system generates a probabilistic data structure (e.g., a Bloom filter) to represent data in a lookup list including multi-token items by hashing items of the lookup list to sets of bit values in a bit vector. The disclosed system classifies text content in a digital document by utilizing a maximum number of tokens from multi-token items in the lookup list to select and compare sets of sequential tokens in the digital document to the probabilistic data structure. The disclosed system also iteratively reduces the number of tokens in sets of sequential tokens for subsequent comparisons. Furthermore, in some aspects, the disclosed system causes a computing device to modify a digital document and / or database operations based on the classifications.

The invention discloses a synchronization method and device for storage metadata, electronic equipment and a storage medium, and relates to the technical field of computers, a consistent Hash algorithm is adopted to realize mapping of data fragments and storage nodes, a monitoring node maintains the relation, a data change event is directionally sent to a corresponding partition, and the data change event is stored in the corresponding partition. Meanwhile, the Bloom filter is used for pre-checking events to filter invalid information, only local related changes are processed, the check value is transmitted, and the memory area is pre-registered, so that the data interaction mode between the nodes is optimized, and the defects of a full-connection mode are avoided; the technical problems that in an existing block storage metadata synchronization mechanism, due to the fact that a full connection mode is not optimized for node scale extension, the network connection number is increased in an exponential level mode, the system overhead is increased, and the cluster expansion capacity is limited can be solved, and the purposes of reducing the system overhead, improving the cluster expansion capacity and improving the cluster expansion efficiency are achieved. And the performance of the distributed block storage system is optimized.

The invention provides a privacy record linking method for low-quality data, which can be applied to the technical field of data fusion entity alignment. The method comprises the following steps: a link participant carries out multiple rounds of negotiation communication based on a link scene, and determines parameter information for subsequently executing a privacy record link; bloom filter coding operation is executed on respective data of link participants, the data missing number is recorded synchronously, and a coding table and a missing table are generated; the link unit analyzes the data of the coding table and the missing table by using the dynamic locality sensitive hash parameter, generates a record partitioning flag bit, and completes record partitioning according to the flag bit to form a partitioning group; a link unit extracts records in the corresponding blocks from the block group, marks the records in the blocks to form record pairs, and calculates a dynamic threshold value and Jaccard similarity of each record pair according to a reference filtering threshold value and a threshold value reduction amount; and in response to the Jaccard similarity being greater than or equal to the dynamic threshold, outputting the corresponding record pair identifier as a result.

The systems and methods disclosed herein orchestrate task execution among autonomous (or semi-autonomous) AI agentic models (“agents”) responsive to a received query by using a hierarchical semantic fingerprinting framework to generate semantic-aware fingerprints for the query and agents. Queries and descriptions of agent capabilities are processed by a series of hierarchical levels using locality-sensitive hash (LSH) functions, where subsequent layers encode more complex semantic information and generate longer hash values. The hash values are aggregated into a semantic fingerprint. A bloom filter cascade uses a series of increasingly accurate hierarchical bloom filters to reject agent fingerprints that differ from the query fingerprint. The remaining agent fingerprints are compared bitwise to the query fingerprint, and those closest to the query fingerprint are selected to generate a routing path for the query. Responses from the selected agents are aggregated into an output that is responsive to the input.

A data lake for enterprise security is created from an asynchronous stream of security events by deduplicating objects and creating metadata related to downstream security functions. Deduplication of objects may be efficiently performed with a bloom filter as objects are ingested into the data lake. The objects may also be augmented with metadata arranged in schemas to facilitate monitoring and use within the data lake.

The invention discloses a digital fingerprint generation method for terminal equipment, and relates to the technical field of digital fingerprint generation, comprising the following steps: S1, collecting multi-dimensional hierarchical features; s2, privacy enhancement; s3, fingerprint anti-regeneration generation is carried out; and S4, fingerprint life cycle management. According to the terminal equipment digital fingerprint generation method, multi-dimensional fine-grained feature acquisition is realized from a non-sensitive layer, a semi-sensitive layer and a high-sensitive layer, a feature weight is optimized through a federated learning framework, fingerprint drift caused by hardware replacement and software updating is effectively resisted, and the user experience is improved. Dynamic Gaussian noise is injected into numerical semi-sensitive features to balance privacy protection and feature effectiveness, 2048-bit key Paillier homomorphic encryption is adopted for high-sensitive features, meanwhile, cross-scene tracking risks are blocked through scenarized sub-fingerprint generation and an original feature immediate destruction mechanism, and the privacy protection and feature effectiveness are improved. And an anti-duplication architecture fusing a Merkle tree and a bloom filter is introduced to greatly reduce the fingerprint coincidence collision rate.

The invention provides a data decoupling centralized query method and system based on MQ message triggering, and relates to the technical field of data query, and the method comprises the steps of receiving a message queue data change message, constructing a field, dividing check nodes by depending on a directed acyclic graph, carrying out data consistency verification, and pushing a transaction identifier to carry out concurrent operation scheduling. And collecting hotspot information by adopting a Bloom filter to adjust data fragmentation and cache distribution, generating a distributed query routing table, and distributing query requests according to the routing table. According to the method, the data consistency is improved, and the query performance is optimized.

The embodiment of the invention provides a batch retrieval method based on a verifiable bloom filter and electronic equipment, and relates to the technical field of data security. The method comprises the following steps: converting a server data set into a key value pair form, and mapping keywords in the data set in the key value pair form into equal repeated codes through an equal repeated code mapping algorithm to obtain an equal repeated code data set; determining a storage position of a second verifiable bloom filter according to a keyword in the equal-repetition-code data set, and storing a data entry corresponding to the keyword in the equal-repetition-code data set and a check code generated based on the data entry to the storage position; the client receives the query keyword set and maps the query keyword set to the first verifiable Bloom filter; and taking each bit of the first verifiable Bloom filter as a selection vector to execute an oblivious transmission protocol to obtain a return result of the server side. The embodiment provided by the invention provides a safe and efficient batch retrieval method based on the verifiable Bloom filter.

The invention provides a data duplicate removal method, electronic equipment and a storage medium. According to the method, smooth data transition is achieved through a double-bloom filter alternating mechanism. In first time period processing, query and addition operations are only executed on a first bloom filter of a current time window. And second time period initialization: creating a new second bloom filter at the first unit time of the second time period as a next window preparation filter. According to the double-writing preheating mechanism, data are written into the first Bloom filter and the second Bloom filter at the same time in the second time period, and the missed judgment problem during window switching is avoided. According to window switching logic, old filter resources are released, the preliminary filter is marked as a new main filter, and monthly / periodic seamless connection is achieved. According to the method, the misjudgment rate is controlled through a double-bloom filter time window switching mechanism, the data duplicate removal accuracy and the system resource utilization rate can be improved, and the method is suitable for the efficient duplicate removal requirement in a mass data collection scene.

The invention provides an enhanced symmetric searchable encryption method and system, and relates to the field of data security and privacy protection. The method comprises the steps that a plaintext document set is symmetrically encrypted, keywords are extracted, and an encryption index token is generated through a pseudo-random function; constructing a reverse index structure, and mapping the token to a document identifier; dynamic insertion, deletion and modification of documents are supported, and indexes are synchronously updated in real time; cipher query is realized by generating a search token, and after the server returns a matching result, the client decrypts the matching result to obtain an original text. According to the system, through multi-keyword Boolean logic query, an index structure is compressed by adopting a Bloom filter, the retrieval efficiency is improved in combination with a jump pointer or a B + tree, a false query strategy is introduced to hide an access mode, and the anti-attack ability is enhanced. Compared with an existing scheme, the method has the advantages that the searchability, the system performance and the privacy protection capability of the encrypted data are improved, and the method is suitable for security data retrieval application in cloud computing, database systems and block chain environments.

A control method includes dividing a source data unit among a plurality of source data units received from a host into a plurality of source chunks; generating a unique value for each of the plurality of source chunks; inserting the unique values into a source bloom filter of the source data unit; calculating a Hamming similarity with a storage bloom filter of each of the plurality of active storage data units for the source bloom filter; selecting an active storage data unit based on the highest Hamming similarity as a target data unit; classifying each of the plurality of source chunks as a first deduplicated chunk, an already deduplicated chunk, or a unique chunk using the unique values; and storing at least one of the first deduplicated chunks in the target data unit, such that spatial locality of each chunk may improve and fragmentation may be alleviated.

Disclosed is system for fulfilling service request using computing nodes. The system comprises a decentralized computing network, comprising computing nodes operating as autonomous agents, configured to implement software framework comprising: client-agent device 5 configured to: receive service request and generate objective therefor, and send objective to agent-device configured to send queries to a machine learning model agent and / or obtain tasks, and list of autonomous agents associated with objective; and processing arrangement configured to: generate directed acyclic graph (DAG) having 10 nodes representing first task and edges representing dependency of first task on second task(s) amongst tasks, and initialize Bloom filter of each node to track completion status of second task(s). Autonomous agent is configured to: identify second task(s), check completion status of second task(s), using Bloom filter of first task, upon completion of second 15 task(s), execute first task, and update Bloom filter.

A method of incorporating a bloom filter (BF) into cost-based bottom-up query optimization includes: receiving a query including a request to join at least two data tables that include a first data table and a second data table; generating, in response to the request, at least two query plans that includes a first query plan including subplans to: create the BF based on the second data table, apply the BF to the first data table for filtering data of the first data table before the first data table is joined with another data table, and join the first data table with the another data table; submitting the at least two query plans to a cost-based bottom-up optimizer to obtain a target query plan; and providing, as a response to the received query, a set of data that is retrieved from a data retrieval system by executing the target query plan.

The invention provides a database index implementation method based on object storage, which is used for solving the problem of efficient retrieval when object storage is used as a database storage engine. According to the method, a special index storage structure is constructed in object storage and comprises an index metadata area and an index data area, and index data is organized through a segmentation management mechanism. Each segment in the index data area comprises a Bloom filter file, an index entry file and a segment abstract file, and efficient organization and access of index data are achieved. According to the method, a batch writing and parallel retrieval strategy is adopted, and a segment merging and garbage collection mechanism is matched, so that the index performance and the storage efficiency are ensured. The method has the beneficial effects that the query performance when the object storage is used as the database is remarkably improved, the storage overhead is reduced, meanwhile, the data consistency is ensured, and a key technical support is provided for realizing a database system based on the object storage.

The invention discloses a method for calculating a threshold multi-party privacy set intersection based on security comparison, which relates to the technical field of privacy protection, and comprises the following steps of: obtaining a encrypted Bloom filter of a private set element of each participant, carrying out Hash operation on the private element of a server, obtaining an encrypted value in the Bloom filter, carrying out homomorphic summation, and obtaining a threshold multi-party privacy set intersection; and calling a security comparison protocol to obtain whether the private element of the server appears in each participant, performing homomorphic summation accumulation on the number of occurrences, then calling the security comparison protocol to compare an accumulation result with a threshold condition, decrypting a comparison result to obtain an element set meeting the threshold condition, and returning the element set to the participant. According to the method, homomorphic encryption and Bloom filter technologies are adopted, calculation of threshold privacy intersection is allowed to be completed in a ciphertext state, a higher privacy protection degree is provided, any privacy information of participants except a result is not leaked, and the method can adapt to different application scenes.

The invention discloses an identification-based lightweight hierarchical trust access authentication method and system, and mainly relates to the technical field of network security. Comprising the following steps: in an offline stage, an electric power registration center generates a private key root bound with an identity for a terminal through identity-based cryptography, a key generation center derives a lightweight work key voucher and pre-calculates verification points of all terminals, and a unified main multi-dimensional Bloom filter is constructed and distributed to a main station; in the online stage, a terminal updates a key and generates a signature only through hash operation, a master station reconstructs candidate verification points after receiving a message, and non-interactive verification is completed by locally querying a master multi-dimensional bloom filter. The method has the advantages that unification of strong identity credibility and extreme light weight of the terminal is achieved, the terminal side only needs efficient Hash operation, the server supports high-concurrency and low-delay authentication through the pre-calculation and Bloom filter technology, and the problem of safe access of massive distributed terminals in the electric power Internet of Things is effectively solved.

The invention discloses a threshold intersection-oriented efficient multi-party privacy set intersection method and system, and belongs to the technical field of privacy computing. Comprising the steps that m participants are initialized, and each participant holds n to-be-solved private data sets; m secret keys are generated, and after the element blinding factor r is selected, the first m-1 secret keys are sent to the corresponding secret keys respectively; each participant carries out partial OPRF value calculation and sends the partial OPRF value to the cloud server through OKVS; after the cloud server carries out OKVS decoding, the OKVS decoding is constructed into a matrix M with the dimension as shown in the specification; the cloud server performs reconstruction calculation according to the matrix M, inserts the matrix M into a Bloom filter and sends the matrix M to the Bloom filter; and querying in the Bloom filter fed back by the cloud server, if the Bloom filter exists, determining that the Bloom filter belongs to the intersection, and if the Bloom filter does not exist, determining that the Bloom filter does not belong to the intersection. Threshold homomorphic encryption does not need to be used for calculation, and the trusted cloud server is introduced to bear part of calculation load, so that the calculation overhead of the protocol is greatly reduced.

The application discloses a multi-service traffic grouping scheduling method for a P4 switch processor core, mainly comprising a feature extraction method based on a statistical interval and a Sketch, comprehensive data packet time correlation and traffic dynamic change, an optimized Bloom filter conflict detection mechanism, improved adaptability and accuracy of the algorithm in a limited resource environment; a decision tree model is combined with scheduling, a decision tree is reconstructed to adapt to a P4 switch operation mode and data processing demand, high-precision service flow classification under line-speed operation is ensured; a grouping scheduling method AD-PIFO is designed based on variable priority, data packet forwarding priority is dynamically adjusted, different service QoS demand is met, and the overall performance and efficiency of the switch are improved. The method is outstanding in high-priority traffic throughput and low-latency performance, and the medium-low-priority traffic throughput is better than that of similar algorithms, is stable and effective under different load conditions, has reasonable additional time overhead of data packet transmission, and is widely used in SDN network optimization data packet scheduling scenes.

The invention relates to a compressed and quantized Internet of Vehicles privacy protection security hierarchical federated learning method, which designs a vehicle end side encryption parameter compression and integer quantization method, a vehicle side completes model training on the basis of local data, and performs flattening, low-rank approximate compression and integer quantization on parameter tensors to obtain a compressed and quantized vehicle end side encryption parameter. And the transmission quantity is obviously reduced under the condition of ensuring the model precision. The quantized parameters are subjected to end-side homomorphic encryption at a vehicle end, and a ciphertext is kept invisible in transmission and aggregation stages. And secondly, a method for setting anonymous authentication batch verification and session key negotiation security encryption between the vehicle end and the fog layer is designed, the concurrent authentication pressure is reduced through a batch verification structure, and the communication security and the access efficiency are improved. And finally, designing a hierarchical ciphertext aggregation updating and Bloom filtering revocation control method, performing homomorphic domain aggregation on ciphertext parameters uploaded by different vehicles by a fog layer, executing one-time decryption by a center end and recovering a floating point model weight, and completing global model updating.

The application discloses a privacy risk detection method, device and equipment of a medical RAG system and a storage medium, wherein a cascade design of a Bloom filter and a MinHash is used to greatly reduce the similarity detection complexity on a large-scale document set, millisecond-level response can be realized under typical hardware conditions, the delay is low, the online medical RAG scene is suitable, not only the text surface similarity is considered, but also medical entities and identifiable information are comprehensively analyzed to form a multi-dimensional risk score, which helps to reduce missed detection and false positives, the detection accuracy is high, the fine-grained risk score and level can be output for each query, resource consumption is controllable, easy to expand, and system integration is friendly.

The application discloses a multi-keyword fuzzy ciphertext retrieval method and system, wherein the method comprises the following steps: decomposing a search request to generate a plurality of query keywords, generating a plurality of query vectors and a plurality of keyword trapdoor functions corresponding to the plurality of query keywords according to the plurality of query keywords; traversing the plurality of keyword trapdoor functions through a plurality of document encryption indexes constructed in advance, respectively calculating the sum of inner products of each document encryption index and each keyword trapdoor function, and obtaining a plurality of similarity values between the plurality of query vectors and the document encryption indexes; comparing the plurality of similarity values, and obtaining a document corresponding to the search request according to a comparison result. Thus, the problems that the retrieval scheme of the related art cannot effectively eliminate the false hits of the Bloom filter, the precision is low, the extracted keyword feature dimension is small, and the feature is degraded after dimension reduction of the keyword feature vector, so that the fuzzy keyword retrieval cannot be realized with high precision and the usability is low are solved.

A memory device is operated using a reduced row address space (e.g., 48k rows per bank vs. a full capacity 64k rows per bank). The unused address space (e.g., 16k rows) is used and mapped by the memory device to store information for at least some of the bad rows. In this manner, many more bad rows may be "repaired" at the expense of a reduction in the capacity of the memory device. The translations of external row addresses to internal physical row addresses may include permuting row addresses within groups of row addresses to avoid bad rows within respective groups of row addresses. A linear filter (e.g., Bloom Filter, XOR filter, etc.) may be used to store information for the translations of external row addresses to internal physical row addresses. The information for the translations may be determined and programmed into the memory device once during manufacturing.

The invention relates to a Bloom filter processing method and device, equipment, a storage medium and a program product. The method comprises the following steps: determining an ordered character string table which is not associated with a global bloom filter from a first storage layer of a log merging tree instance to obtain an ordered character string table set; under the condition that the number of the ordered character string tables in the ordered character string table set is greater than a number threshold value, selecting an ordered character string table subset from the ordered character string table set according to the number threshold value; associating a global bloom filter for each ordered character string table in the ordered character string table subset, and updating the associated global bloom filters according to the ordered character string tables; and entering the step of determining the ordered character string table which is not associated with the global bloom filter from the first storage layer of the log merge tree instance to obtain an ordered character string table set so as to associate the ordered character string table which is not associated with the global bloom filter with the global bloom filter in batches. By adopting the method, the query efficiency can be improved.

The invention relates to a message broadcast deduplication method and device based on a Bloom filter and computer equipment, and the method comprises the steps: judging whether a message is processed or not based on a local message state record after a downstream node receives the message and corresponding path information sent by an upstream node; the message state is recorded as a counting Bloom filter storing the processed message; when it is judged that the message is not processed, the message state record is correspondingly updated; and determining a forwarding target corresponding to the downstream node according to the path information so as to broadcast a message to the downstream. Through the application, the message state record table can be continuously and stably maintained through the counting bloom filter, the node is prevented from repeatedly processing the message, and the downstream node can optimize the broadcast path according to the corresponding path information of the message, thereby avoiding message redundancy in the broadcast process, and improving the broadcast efficiency. And a stable and effective message broadcast deduplication method is realized through a two-layer deduplication mechanism.

The application relates to the field of privacy protection query schemes, in particular to an indoor multi-user personalized spatial keyword privacy query method based on a GIS enhanced index tree. In view of the demand of multi-user spatial keyword privacy query in an indoor environment, a GISEnhancedIndexTree index structure is constructed by fusing Hilbert coding, an R tree, an encrypted Bloom filter and an EASPE encrypted vector, and spatial and text information of indoor objects are encrypted and indexed. During query, a user request is encrypted and processed into a trapdoor, a server performs spatial, floor and keyword pre-filtering in the index tree according to the trapdoor, calculates the similarity of encrypted vectors by means of EASPE, and returns the encrypted result to the user after threshold filtering, score aggregation and sorting. Meanwhile, a comprehensive position vector construction method and a privacy protection coordinate conversion method based on a local coordinate system are designed to guarantee the privacy safety of the whole data flow. The method can effectively protect the privacy of users and improve the efficiency and accuracy of indoor multi-user personalized spatial keyword query.

The application discloses a minimum community search method, device and system and a storage medium, and is applied to the field of minimum community search, and comprises the following steps: receiving an adjacent index and a core tree index which are encrypted by a BGN algorithm and are sent by a data end; wherein the data end constructs a corresponding Bloom filter array for each tree node in the core tree index; when receiving a query trapdoor token generated by a client, locating an initial community in the core tree index according to the Bloom filter array; generating a target community based on the initial community and the adjacent index by using an array comparison protocol; and sending the target community to the client, so that the client decrypts the target community to obtain a target result. The method avoids the privacy leakage problem caused by the minimum community search in a cloud server in the prior art by encrypting the minimum community search, and protects data security.

The invention discloses a privacy set intersection method and system supporting load interval screening, and the method comprises the steps: determining a Bloom filter and a hash function, generating a homomorphic encryption key pair by a first participant, setting a threshold parameter and a load interval by a second participant, and generating a symmetric encryption key, a master secret and a random number; the first participant obtains a ciphertext and sends the ciphertext to the second participant, the second participant generates a homomorphic ciphertext, and the second participant encrypts elements, a load value and a load interval boundary value of the homomorphic ciphertext and sends the elements, the load value and the load interval boundary value to the first participant together in combination with a threshold parameter; the master secret is recovered through threshold judgment; the first participant decrypts and screens out the intersection element ciphertext meeting the load interval, and sends the intersection element ciphertext to the second participant; and outputting the intersection element meeting the load interval condition through decryption. According to the invention, intersection calculation of load values in a specific range is realized.