826 results about "Bloom filter" patented technology

According to an embodiment of the invention, cache management comprises maintaining a cache comprising a hash table including rows of data items in the cache, wherein each row in the hash table is associated with a hash value representing a logical block address (LBA) of each data item in that row. Searching for a target data item in the cache includes calculating a hash value representing a LBA of the target data item, and using the hash value to index into a counting Bloom filter that indicates that the target data item is either not in the cache, indicating a cache miss, or that the target data item may be in the cache. If a cache miss is not indicated, using the hash value to select a row in the hash table, and indicating a cache miss if the target data item is not found in the selected row.

A method, system, and computer usable program product for parallel data redundancy removal are provided in the illustrative embodiments. A plurality of values is computed for a record in a plurality of records stored in a storage device. The plurality of values for the record is distributed to corresponding queues in a plurality of queues, wherein each of the plurality of queues is associated with a corresponding section of a Bloom filter. A determination is made whether each value distributed to the corresponding queues for the record is indicated by a corresponding value in the corresponding section of the Bloom filter. The record is identified as a redundant record in response to a determination that each value distributed to the corresponding queues for the record is indicated by a corresponding value in the corresponding section of the Bloom filter.

A method and system for generating a bloom filter by mapping into respective groups each of a plurality of initial keys according to a first hash function and mapping each group hashed key into a bloom filter using k respective hash functions.

The present invention relates to a method and apparatus based on Bloom filters for detecting predefined signatures (a string of bytes) in a network packet payload. A Bloom filter is a data structure for representing a set of strings in order to support membership queries. Hardware Bloom filters isolate all packets that potentially contain predefined signatures. Another independent process eliminates false positives produced by the Bloom filters. The system is implemented on a FPGA platform, resulting in a set of 10,000 strings being scanned in the network data at the line speed of 2.4 Gbps.

A system and method for detecting duplicate checks during processing. The duplicate detection may be performed by a financial institution, such as a bank. The method may be implemented on a computer based system. The duplicate detection method may be automated. The method may be applied to incoming check files prior to processing of the check data to prevent processing of duplicate checks. The system and method may use a function, such as a hash function, to perform the duplicate detection. Other functions, such as a Bloom filter which may use multiple hash functions, may be used to perform the duplicate detection.

The present invention relates to a method and system of performing parallel membership queries to Bloom filters for Longest Prefix Matching, where address prefix memberships are determined in sets of prefixes sorted by prefix length. Hash tables corresponding to each prefix length are probed from the longest to the shortest match in the vector, terminating when a match is found or all of the lengths are searched. The performance, as determined by the number of dependent memory accesses per lookup, is held constant for longer address lengths or additional unique address prefix lengths in the forwarding table given that memory resources scale linearly with the number of prefixes in the forwarding table. For less than 2 Mb of embedded RAM and a commodity SRAM, the present technique achieves average performance of one hash probe per lookup and a worst case of two hash probes and one array access per lookup.

A system and method for performing biological sequence similarity searching is disclosed. This includes a programmable logic device configured to include a pipeline that comprises a matching stage, the matching stage being configured to receive a data stream comprising a plurality of possible matches between a plurality of biological sequence data strings and a plurality of substrings of a query string. The pipeline may further include a ungapped extension prefilter stage located downstream from the matching stage, the prefilter stage being configured to shift through pattern matches between the biological sequence data strings and the plurality of substrings of a query string and provide a score so that only pattern matches that exceed a user defined score will pass downstream from the prefilter stage. The matching stage may include at least one Bloom filter.

The Computationally Efficient Transfer Processing and Auditing Apparatuses, Methods and Systems (“CETPA”) transforms transaction record inputs via CETPA components into matrix and list tuple outputs for computationally efficient auditing. A blockchain transaction data auditing apparatus comprises a blockchain recordation component, a matrix Conversion component, and a bloom filter component. The blockchain recordation component receives a plurality of transaction records for each of a plurality of transactions, each transaction record comprising a source address, a destination address, a transaction amount and a timestamp of a transaction; the source address comprising a source wallet address corresponding to a source digital wallet, and the destination address comprising a destination wallet address corresponding to a destination virtual currency wallet; verifies that the transaction amount is available in the source virtual currency wallet; and when the transaction amount is available, cryptographically records the transaction in a blockchain comprising a plurality of hashes of transaction records. The Bloom Filter component receives the source address and the destination address, hashes the source address using a Bloom Filter to generate a source wallet address, and hashes the destination address using the Bloom Filter to generate a destination wallet address. The Matrix Conversion component adds the source wallet address as a first row and a column entry to a stored distance matrix representing the plurality of transactions, adds the destination wallet address as a second row and column entry to the stored distance matrix representing the plurality of transactions, adds the transaction amount and the timestamp as an entry to the row corresponding to the source wallet address and the column corresponding to the destination wallet address; and generate a list representation of the matrix, where each entry in the list comprises a tuple having the source wallet address, the destination wallet address, the transaction amount and the timestamp.

Packets are routed in a wireless mesh network by assigning a probability of packet retransmission based on certain factors which may include the number of hops the packet has made, whether the destination is within a certain number of known hops of the receiving node, the number of nearby nodes, whether the receiving node is a member of a past known route to the destination and / or whether a nearby neighbor of the receiving node is a member of a past known route to the destination. Once a probability is assigned, a random number generator may be used to make a decision between retransmitting the packet and dropping the packet. Signal quality may also be used as a factor in the probability of retransmission. In a preferred embodiment, Bloom filters are used to store information in the header of a data packet containing the identifiers of forwarding nodes and the identifiers of nodes on a previously-successful route from the source to the destination.

The present invention relates to a method and apparatus based on Bloom filters for detecting predefined signatures (a string of bytes) in a network packet payload. A Bloom filter is a data structure for representing a set of strings in order to support membership queries. Hardware Bloom filters isolate all packets that potentially contain predefined signatures. Another independent process eliminates false positives produced by the Bloom filters. The system is implemented on a FPGA platform, resulting in a set of 10,000 strings being scanned in the network data at the line speed of 2.4 Gbps.

Systems and methods are disclosed for detecting covert DNS tunnels using n-grams. The majority of legitimate DNS requests originate from network content itself, for example, through hyperlinks in websites. So, comparing data from incoming network communications to a hostname included in a DNS request can give an indication on whether the DNS request is a legitimate request or associated with a covert DNS tunnel. This process can be made computationally efficient by extracting n-grams from incoming network content and storing the n-grams in an efficient data structure, such as a Bloom filter. The stored n-grams are compared with n-grams extracted from outgoing DNS requests. If n-grams from an outgoing DNS request are not found in the data structure, the domain associated with the DNS request is determined to be associated with a suspected covert DNS tunnel.

A computer-implemented method for managing concurrent transactions includes recording locations written by a first transaction in a first Bloom filter, recording locations to be read by a second transaction in a second Bloom filter, and performing one of a cancellation or a commission of the second transaction based on an intersection of the first Bloom filter and the second Bloom filter.

Methods for performing exact search string matches in the data plane of a network processor. The methods employ a two-stage string search mechanism to identify the existence of a search string from a set S in a packet payload. A first pre-processing stage identifies a potential search string match and a second analysis stage determines whether the first stage match corresponds to an exact string match. The first stage is implemented using hash values derived from at least one of search strings in set S or sub-strings of those search strings. In one embodiment, a plurality of Bloom filters are used to perform the first pre-processing stage, while in other embodiments various CAM-based technique are used. Various TCAM-based schemes are disclosed for performing the second analysis stage. The methods enable packet payloads to be searched for search strings at line-rate speeds.

The invention discloses a distributed file system metadata management method facing to high-performance calculation. The method comprises the following steps of: 1) establishing a catalogue metadata server cluster, a file metadata server cluster and an object storage server cluster; 2) establishing a global counting-based bloom filter in the catalogue metadata server cluster; 3) when the operation request of a client side arrives, skipping to execute step 4) or 5); 4) enabling the catalogue metadata server cluster to respond to the catalogue operation request of the client side to manage the catalogue metadata; and 5) enabling the file metadata server cluster to respond to the file operation request of the client side to manage the file metadata data. According to the distributed file system metadata management method disclosed by the invention, the metadata transferring problem brought by catalogue renaming can be effectively solved, and the distributed file system metadata management method has the advantages of high storage performance, small maintenance expenditure, high load, no bottleneck, good expansibility and balanced load.

An integrated circuit, and method of reviewing values of one or more signals occurring within that integrated circuit, are provided. The integrated circuit comprises processing logic for executing a program, and monitoring logic for reviewing values of one or more signals occurring within the integrated circuit as a result of execution of the program. The monitoring logic stores configuration data, which can be software programmed in relation to the signals to be monitored. Further, the monitoring logic makes use of a Bloom filter which, for a value to be reviewed, performs a hash operation on that value in order to reference the configuration data to determine whether that value is either definitely not a value within the range or is potentially a value within the range of values. If the value is determined to be within the set of values, then a trigger signal is generated which can be used to trigger a further monitoring process.

Implementation of a Bloom filter using multiple single-ported memory slices. A control value is combined with a hashed address value such that the resultant address value has the property that one, and only one, of the k memories or slices is selected for a given input value, a, for each bank. Collisions are thereby avoided and the multiple hash accesses for a given input value, a, may be performed concurrently. Other embodiments are also described and claimed.

Methods, systems, and machine-readable media are disclosed for searching a corpus of information by utilizing a Bloom filter for caching query results. According to one aspect of the present invention, a method of caching information from a corpus of information can include populating one or more Bloom filters with a plurality of bits representative of information in the corpus of information. A search request can be received identifying requested information from the corpus of information. One or more bits in the filter(s) associated with the requested information can be checked and the requested information can be retrieved from the corpus of information based on results of said checking. Furthermore, the filter(s) can be used to determine which information to make available to a particular user in a system where certain information is associated with or access is limited to certain users or groups of users.

Symmetric Connection Detection (SCD) is a method of detecting when a connection has been fully established in a resource-constrained environment, and works in high-speed routers, at line speed. Many network monitoring applications are only interested in connections that become fully established, so other connection attempts, such as port scanning attempts, simply waste resources if not filtered. SCD filters out unsuccessful connection attempts using a simple combination of Bloom filters to track the state of connection establishment for every flow in the network. Unsuccessful flows can be filtered out to a very high degree of accuracy, depending on the size of the bloom filter and traffic rate. The SCD methodology can also easily be adapted to accomplish port scan detection, and to detect or filter other types of invalid TCP traffic.

Systems and methodologies are described that facilitate identifying peers based upon encoded signals during peer discovery in a peer to peer network. For example, direct signaling that partitions a time-frequency resource into a number of segments can be utilized to communicate an identifier within a peer discovery interval; thus, a particular segment selected for transmission can signal a portion of the identifier, while a remainder can be signaled based upon tones communicated within the selected segment. Moreover, a subset of symbols within the resource can be reserved (e.g., unused) to enable identifying and / or correcting timing offset. Further, signaling can be effectuated over a plurality of peer discovery intervals such that partial identifiers communicated during each of the peer discovery intervals can be linked (e.g., based upon overlapping bits and / or bloom filter information).

In one embodiment of the invention, service providers generate bloom filters with the user ID codes of registered users and exchange the bloom filters with one another. In response to a request to locate a first user, a first service provider will query its own registration database to determine if the first user is registered with the first service provider. If the first user is not registered with the first service provider, then the first service provider will query its bloom filters to identify other service providers with which the first user may be registered. A positive response from a bloom filter indicates that the first user may or may not be registered with the service provider associated with that bloom filter, and a negative response indicates with certainty that the first user is not registered with the service provider associated with that bloom filter.

Embodiments are directed to perfect physical garbage collection (PPGC) process that dynamically estimates duplicate containers using a Bloom filter-based dead vector by scanning an index containing a mapping of fingerprints to a container ID for a plurality of containers; returning, for each fingerprint, a fingerprint sequence associating each fingerprint with a respective unique container ID, wherein a last entry of the sequence is preserved and the remaining entries are considered duplicates; and maintaining a duplicate array of counts of the duplicates indexed by container IDs, and wherein the duplicate array comprises a duplicate counter that keeps track of a number of live duplicated segments for each container, and further wherein a live segment is a live duplicate segment if a segment with a same fingerprint exists in another container with a higher container ID.

A network address associated with a packet is obtained at a search engine of a network device. The search engine includes a plurality of Bloom filters that represent prefixes of respective lengths in the routing table. Respective Bloom filters are applied to respective prefixes of the network address to determine a set of one or more prefixes for which a match potentially exists in the routing table. A number of accesses to the memory are performed using prefixes in set of prefixes, beginning with a longest prefix and continuing in decreasing order of prefix lengths until a matching entry is found in the routing table, and routing information for the packet is retrieved. If the number of performed memory accesses exceeds a threshold, the routing table is adapted to reduce a number of memory accesses to be performed for subsequent packets associated with the network address.

A system that determines whether a pattern of data elements was observed in a stream of data. During operation, the system receives a query which seeks to determine whether the pattern of data elements was observed in the stream of data. In response to the query, the system performs a number of lookups in an enhanced Bloom filter to determine whether the pattern of data elements was observed in the stream of data, wherein the enhanced Bloom filter includes multiple instances of a Bloom filter, and wherein each instance of the Bloom filter is associated with a different time interval. If so, the system generates a notification that the pattern of data elements was observed.

Compressed Bloom filters that act as a message as well as a data structure provide smaller false positive rates, reduced bits broadcast and / or reduced computational overhead in distributed Web proxy servers and other distributed networks.

A network node (NB1) located within a domain is adapted to receive, from another node, a packet having an in-packet Bloom filter or Bloom filter equivalent encoding information about a route within the domain. The node reversibly modifies the in- packet Bloom filter or Bloom filter equivalent in a manner which is linear with respect to the operation used to add links to the Bloom filter or Bloom filter equivalent. The node then forwards the packet with its header containing the modified Bloom filter or Bloom filter to another node (NA1). The invention allows secure Bloom filter-based routing in a domain (Domain B), while requiring that only routers (NB1) at the domain boundary are secure routers. Other routers (NB2, NB3, NB4) in the domain may operate conventionally, and may be secure routers or insecure routers. The modification may be a bit permutation.

A system for index searching comprises a first index, a second index, a processor, and a memory. The processor is configured to determine, using a Bloom filter for the second index of fingerprints, whether the Bloom filter indicates that an entry corresponding to a segment has been previously stored in the second index. The processor is further configured to, in the event that the Bloom filter indicates that an entry corresponding to the segment has been previously stored in the second index, retrieve a second index entry associated with the segment that has been previously stored. The memory is coupled to the processor and configured to provide the processor with instructions.

In one embodiment, IP lookup into a routing table having prefixes of different prefix lengths is performed by hashing a candidate prefix value to generate a plurality of hash values, where m seed hash values are generated by applying m seed hash functions and one or more additional hash values are generated by combining two or more of the seed hash values in different ways, e.g., using a bit-wise XOR function. The hash values are used to perform membership probes into a Bloom filter programmed with the prefixes corresponding to all of the different prefix lengths in the routing table without having to expand any of the prefixes programmed into the Bloom filter.

A hierarchical data structure of digested payload information (e.g., information within a payload, or information spanning two or more payloads) allows a payload excerpt to be attributed to earlier network flow information. These compact data structures permit data storage reduction, while permitting efficient query processing with a low level of false positives. One example of such a compact data structure is a hierarchical Bloom filter. Different layers of the hierarchy may correspond to different block sizes.

The invention provides a method for implementing transaction query in a distributed database on the basis of a block chain. The method comprises the following steps: step S101, designing a Bloom filter, and finding out a candidate block which comprises an assigned transaction ID according to the designed Bloom filter; step S102, traversing and screening the candidate block, finding out a specific block, which comprises a transaction ID, of target transaction, and then constructing a Verify Message; step S103, verifying the completeness of a transaction path of the block by analysis of the Verify Message; and step S104, returning corresponding information according to a verify result, and implementing query of optional transaction in the distributed database on the basis of the block chain. By the method, query of the optional transaction in the distributed database on the basis of the block chain is realized, and the utilization rate of data of the block chain is increased.