50 results about "HashClash" patented technology

The invention relates to the technical field of code caching, and provides a dynamic binary translation caching method and system based on hotspot discovery, and the dynamic binary translation caching method comprises the following steps: S1, taking a virtual address of a client as a key value of a Hash conflict ring structure; s2, in a certain detection, if it is detected that the same translation block is hit for the second time according to the virtual address of the client and the program counter, and the number of hit translation blocks reaches a hit threshold value, triggering a hotspot transfer thread to update a hotspot counter corresponding to translation block information, pointing a head pointer of the hash bucket to the translation block information with the highest hotspot counter value; and S3, two groups of alternately used continuous memory spaces are adopted, translation codes of the old memory space are recovered through copy-on-write, new translation codes are written into the new memory space, and the translation codes of the old memory space are asynchronously migrated to the new memory space.

The invention discloses a hash table processing method and device, a query method and device and a storage medium. The hash table processing method comprises the steps of determining a target hash bucket of a to-be-stored key-value pair in response to a storage request of the to-be-stored key-value pair, and judging whether a hash conflict exists or not according to a storage state of the target hash bucket of the to-be-stored key-value pair; determining the storage position of the key value pair to be stored according to the position state of the key value pair stored in the target hash bucket recorded by the bitmap and / or the storage state of other hash buckets except the target hash bucket under the condition that the hash conflict exists; and storing the to-be-stored key value pair in the determined storage position, recording the position state of the key value pair by using a bitmap, and optimizing the storage position of the to-be-stored key. According to the method, the bitmap and the hash bucket are combined, the storage resource of the hash table can be effectively utilized, the storage position of the key value pair is reasonably determined without expanding the hash bucket, and the utilization rate of the storage resource is improved.

The invention discloses a data security union set processing method and system, belongs to the technical field of data management, and solves the problems that hash conflicts are inherent defects of a hash algorithm and cannot be completely avoided, fragment parallelism can be damaged by introduction of cross-fragment retrieval, merging complexity is remarkably increased, and data security is poor. And therefore, actual associated data may be dispersed in different fragments due to hash conflicts in fragment number matching, and cross-fragment retrieval is needed. Comprising the following steps: dynamically fragmenting according to the data popularity of a primary key; through dynamic fragmentation, distributed bloom filter conflict pre-detection and local redirection query technologies, multi-thread parallel processing and real-time load balancing optimization are combined, the cross-fragmentation retrieval proportion caused by hash conflicts is limited, the hash conflict rate is reduced while the fragmentation parallelism is guaranteed, super-large data sets are merged, and the retrieval efficiency is improved. And the problem that in fragment number matching, actual associated data are possibly dispersed in different fragments due to Hash conflicts, and cross-fragment retrieval is needed is avoided.

The present invention relates to an optimized access method and access system for a hash table in an OpenGauss hash connection. The method comprises the steps of constructing a hash table, completing the filling of a hash bucket array and a hash collision array; traversing the hash bucket array to detect whether a hash collision has occurred in the data stored at each location; and filling prompt information with the free space of the location information stored in the hash bucket array and the hash collision array based on the detection result. The method combines the advantages of the chain address method and the open addressing method, optimizes the implementation of the hash table from multiple perspectives, and distinguishes whether each conflicting data uses the chain address method or the open addressing method by filling in prompt information, thereby greatly improving the access performance of the hash table. In addition, the prompt information provided by the method can be used to determine whether a hash collision has occurred in the data without accessing the HashNext array, thereby improving the memory access and hash connection performance of the hash table.

The invention provides a character string coding and mapping method, system and device, a medium and a program product. The method comprises the steps that a to-be-coded character string is calculated through a self-defined coding polynomial to obtain a coding value corresponding to the character string; for each character string, allocating an index address to store a storage path corresponding to the character string in a storage space of a preset specification indicated by the index address, and storing the corresponding character string in a storage space indicated by the storage path; and storing the coded value of each character string and the corresponding index address in a lookup table by taking the coded value as an index so as to indicate a mapping relationship among the character string, the coded value, the index address and the storage path. The character string is coded through the polynomial, the prime number with the good hash characteristic serves as the polynomial cardinal number, the distribution uniformity of coded values is improved, and therefore the hash conflict probability is reduced; the realization of unique coding of the character string under the condition that the coding space is limited is ensured, and the system stability and the retrieval accuracy are improved.

The application discloses a unique hash serial number generation method and system, solves the problem of a large hash value range interval, and improves the space utilization rate and data query efficiency of an object with a string as an index during storage. The technical scheme is as follows: a string is processed to generate a hash serial number which is self-incremented from 0, the hash serial number is used as an index to replace the original storage data structure which needs a string as an index; a cblock_map data cache structure is used, the cblock_map uses a hash value of a hash item as an index when storing the hash item, and contains information such as a string and a hash serial number. In the running process of the method, the cblock_map memory structure is stable, all possible spaces are prepared in advance in the construction method, and there is no scenario of recreating an object and copying data after the space is full. The cblock_map provides efficient and stable read and write capabilities, supports O(1) efficiency of the index storage position of the hash value, and even in the case of hash conflict, the query time can be controlled in a constant range through the way of list traversal.

According to the updating method of the FC switch monitoring lookup table based on the LFU, the storage of the monitoring port lookup table is realized by adopting the idea of Robin Hood Hashing, and according to the strategy, a key with a longer detection distance hijacks the position of a key with a shorter detection distance when a Hash conflict occurs during insertion, so that the query performance is optimized, and the updating method is suitable for a high-performance network scene; secondly, modular design is adopted, message IDs existing in a monitoring port lookup table are stored in a grouped mode according to the access frequency through a dictionary and a double-linked list structure, and global sorting is avoided; thirdly, when the message ID stored in the monitoring port query table is full and updated, a double-layer elimination strategy is adopted, the message ID which is least recently used can be efficiently positioned, all frequencies do not need to be traversed, rapid updating is achieved, the method is suitable for a high-concurrency or large-scale FC network communication scene, and therefore the judgment efficiency of the target motion state is improved.

This invention discloses a DDoS attack detection method based on heteroscedastic unscented Kalman filtering. Addressing the issues of limited resources in edge networks and low anomaly detection accuracy and high latency in low signal-to-noise ratio environments, this invention utilizes Sketch for full-scale, unsampled traffic feature extraction in the programmable data plane and reports observations to the control plane through differential processing. In the control plane, a third-order state-space model incorporating instantaneous traffic rate, rate of change, and acceleration is innovatively constructed. An additive heteroscedastic noise model is proposed to adaptively handle Sketch hash collisions and traffic shot noise, and unscented Kalman filtering is employed for state estimation. Simultaneously, an intelligent hybrid control mechanism is introduced to overcome filtering lag, and finally, NIS and CUSUM are combined for dual-modal anomaly detection, outputting the system state. This invention enables high-precision real-time detection of DDoS attacks with extremely low resource overhead, effectively solving the trade-off between measurement and detection in existing technologies.

The invention provides a data transmission method, device, equipment and product, and relates to the technical field of communication networks. The method comprises: receiving a first data packet, the first data packet being a current to-be-transmitted data packet in a logic frame, the first data packet comprising first equalization sequence information; determining a value and a first Hash conflict number corresponding to the first balancing sequence information, the value being used for indicating transmission of the first data packet according to a load balancing mode; and releasing the corresponding value when a release condition is satisfied, the release condition being associated with the number of first hash conflicts and whether the first data packet is a tail packet of the logic frame. According to the mode, the storage space overhead of the value is reduced.

The invention provides a data storage method, device and equipment, and the method comprises the steps: determining a target bit number P for newly-added to-be-stored data in a bit cutting hash data structure; the bit cutting hash data structure comprises a plurality of hash buckets, and the target bit number P is used for enabling the to-be-stored data and the stored data in the bit cutting hash data structure not to have hash conflicts; determining a BSS value of each bit based on the stored data and the to-be-stored data, and determining P feature bits based on the BSS value of each bit; based on the corresponding numerical values of the P feature bits in the to-be-stored data, determining a corresponding target hash bucket of the to-be-stored data in the bit cutting hash data structure; and storing the to-be-stored data in the target hash bucket. According to the scheme, the accurate matching performance of the data can be improved.

The invention provides a method and device for identifying duplicated data based on hierarchical hash lookup, and the method comprises the steps: executing two independent hash operations in parallel for input data to obtain two lookup addresses, and constructing a lookup structure consisting of a plurality of levels of mutually independent hash tables and a residual table. The method comprises the following steps: firstly, carrying out double-hash search on data in a first-level hash table, and cascading the data to a next-level hash table if a conflict occurs until all hierarchical tables are accessed; and for the data which still conflict, sequentially searching in the residual table. According to the method, the probability of Hash conflicts is effectively reduced by adopting the multi-stage independent Hash table and a double-Hash lookup mechanism; more importantly, most of the search delays are limited in a predictable limited clock period by the hierarchical pipeline design, so that the search delay under the worst condition is obviously optimized, and an extremely high system throughput rate is realized in combination with multi-pass parallel.

The application discloses a kind of spaceborne high-speed hybrid hash routing lookup method and system.For improving routing lookup efficiency, pre-processing is set in the front end of core routing lookup module, and multi-channel IP message data stream arbitration is combined into two ways, routing lookup is carried out simultaneously in two ways, and finally the result of routing lookup is distributed to corresponding channel;Secondly, the hybrid hash routing lookup mode of routing lookup process is combined with segmented parallel lookup, multiple hash bucket parallel lookup and multiple hash algorithm lookup, and each result is filtered and output according to table item priority after obtaining the query result.The application combines the characteristics of hash table lookup time complexity O (1), has good query performance, effectively reduces the probability of hash conflict, greatly improves the probability of successful storage of hash table item on the basis of saving hardware storage resources, greatly improves the query accuracy, and solves the demand of high-speed IP routing lookup on satellite product.

The invention provides a hash conflict processing method and device, equipment, a storage medium and a product. The method comprises the following steps: acquiring a conflict element and an original Hash slot of the conflict element in a Hash table, and detecting a Hash slot in an adjacent window on the rear side of the original Hash slot backwards by taking the original Hash slot as a starting point; if the empty Hash slot does not exist in the adjacent window on the rear side, moving elements stored in the Hash slot in the Hash table to enable the empty Hash slot to exist in the adjacent window on the rear side, and inserting conflict elements into the empty Hash slot; configuring a preset identifier for the original Hash slot and the empty Hash slot into which the conflict element is inserted, and generating an adjacent bitmap of the original Hash slot according to the configured preset identifier; and when a conflict element search instruction is received, reading the adjacent bitmap, determining a to-be-matched hash slot configured with a preset identifier, and determining the to-be-matched hash slot of which the stored element is the same as the conflict element as a target hash slot. According to the method, the cache hit rate and the search efficiency can be improved when the Hash conflict is processed.

The invention discloses a distributed model training on-network convergence method based on a programmable switch, which comprises the following steps of: calculating a register slot index position when a model gradient package is converged on the programmable switch by adopting a plurality of hash functions; the problem of Hash collision is solved by dividing the available memory of the programmable switch into a plurality of Hash intervals, and compared with the existing mode of only using a single Hash function, the proportion of Hash collision during convergence of the model gradient packets is effectively reduced, the number of the model gradient packets converged on the network on the programmable switch is increased, and the convergence efficiency of the model gradient packets is improved. Therefore, the model gradient flow sent by the programmable switch to the model convergence server and the competition of the model gradient flow on the limited bandwidth of the inlet of the model convergence server are reduced, and the purposes of accelerating distributed model training and reducing the network flow load of the data center are achieved.

This invention discloses a method and system for local differential privacy aggregate value data aggregation based on sketches and sampling, belonging to the field of privacy computing and big data analysis technology. The method includes: initializing and disclosing sketch structure parameters on the server side; constructing aggregate value data on the user side, randomly selecting counter indices and determining the hash collision set; randomizing the hash collision set elements on the user side according to the aggregation function type, calculating the hit rate and performing adaptive pruning; reporting triples on the user side; and accumulating and updating the sketch counter on the server side, calculating the aggregated statistics through median estimation. This invention replaces traditional multi-element perturbation with sketch sampling and lightweight reporting, without requiring additional privacy budget consumption, effectively solving the noise accumulation problem, significantly reducing computational and communication overhead, and achieving a balance between privacy protection and statistical utility.

The invention provides a message classification method and device, and the method comprises the steps: taking all preset quintuples as a statistical sample, carrying out the calculation, and obtaining the information entropy of each quintuple byte, and enabling the preset quintuples to be quintuples configured by network nodes; a first Hash result corresponding to a preset quintuple is obtained through calculation according to a preset Hash algorithm, first configuration information is stored with the first Hash result as an address, the Hash result comprises two necessary bytes, each necessary byte is obtained through bitwise exclusive or of a plurality of quintuple bytes, and the first configuration information is stored according to the bitwise exclusive or of the quintuple bytes. Two quintuple bytes with the maximum information entropy participate in operation of two necessary bytes respectively; a second hash result corresponding to an actual quintuple is obtained through calculation according to a preset hash algorithm, second configuration information is inquired with the second hash result as an address, and the actual quintuple is a quintuple carried by the message; and if the second configuration information is queried, carrying out message classification according to the second configuration information. By means of the message classification method and device, the hash conflict situation during message classification can be reduced.

A network device includes a storage device, a hardware-accelerated forwarding circuit, and a network processing unit (NPU). The storage device stores a software flow table, where the software flow table includes a plurality of table entries. The hardware-accelerated forwarding circuit receives a first packet from a network, obtains a first hash value of the first packet, and determines that the first hash value encounters a hash collision in a hardware flow table. The NPU receives the first packet from the hardware-accelerated forwarding circuit, and deals with forwarding of the first packet according to forwarding information recorded in a table entry when the first packet hits the table entry of the software flow table.

This application proposes a keyword private information retrieval method and device based on a hierarchical dynamic constant weight code, including the following steps: constructing a multidimensional hypercube storage unit and a collision-resistant hash function through the server, mapping the coordinates of each data in the hypercube storage unit to a unique coordinate constant weight code in a hierarchical mapping manner; the client obtains a query vector and sends it to the server; the server expands the query vector to obtain an extended query vector, obtains the query result based on the extended query vector, and returns the query result to the client; the client decrypts the result and obtains the data to be retrieved based on the key-value pair. This solution uses a multidimensional hypercube storage unit and a collision-resistant hash function to perform data mapping and storage, reducing hash conflicts and avoiding global reconstruction, and optimizes the index update time complexity through hierarchical dynamic constant weight codes to achieve dynamic incremental expansion.

Methods and systems for estimating the frequency table of a data set stored in a probabilistic data structure such as a counting Bloom filter or Count-Min sketch. The system observes the distribution of values in the counters of the data structure (e.g., how many counters have value 0, 1, 2, etc.) Using these observed values, the system iteratively calculates the number of unique items in the data set that appear with specific frequencies. The method accounts for hash collisions by modeling the probability that a counter value results from a single item versus multiple colliding items, allowing for the reconstruction of the item frequency distribution without querying individual elements.

The invention provides a network data processing method and device. The method comprises the following steps: when a data packet arrives at network equipment, extracting an original quintuple of the data packet; when the target index table does not contain the original quintuple, generating a target random number for the original quintuple, and recording a corresponding relation between the original quintuple and the target random number in the target index table; constructing a pseudo-random quintuple according to the original quintuple and the target random number; and forwarding the data packet, and recording the corresponding relationship between the pseudo-random quintuple and the forwarding path information of the data packet in the target hash table. According to the method, a random number is newly added on the basis of a quintuple to construct a pseudo-random quintuple, hash calculation is carried out on the basis of the pseudo-random quintuple, the randomness of the pseudo-random quintuple is enhanced through the random number, it is ensured that different inputs generate different hash values, the hash conflict probability is effectively reduced, and the performance and efficiency of a hash table are improved.

The invention discloses a data deduplication method and system applied to automatic acquisition of quality information, belongs to the technical field of data processing, and aims to solve the technical problem of how to overcome hash conflicts and data sensitivity in data deduplication. Comprising the following steps: selecting a to-be-crawled website and a theme module based on a predefined webpage crawling duplicate removal rule; periodically carrying out website data crawling, and recording the website data crawled by each website into a website library table corresponding to the website; performing in-site data deduplication operation on the website data under each website based on the title and the release time, performing inter-site data deduplication operation on the website data among different websites based on the hash value of the website data, and writing the website data of each website after the inter-site data deduplication operation and the hash value of the website data into a temporary library; and performing data deduplication operation on the website data in the temporary library based on the hash value of the website data, and writing the deduplicated website data and the hash value of the website data into a formal library.

The invention relates to the field of network data processing, discloses a method and a device for collecting large-scale IP address data, provides two efficient large-scale IP address statistical algorithms, and aims to balance time efficiency and memory consumption. According to the method, the sparsity characteristic of IP address statistics is fully considered, and optimization is achieved by dynamically adjusting the mapping relation between the layered memory blocks. In one method, a double-layer structure is adopted, and each layer is composed of a plurality of memory blocks with a fixed number. Each memory block comprises 256 elements, the size of a single element is 8 bytes, and the memory block is adaptive to a 64-bit system. Compared with a built-in hash table, the method has the advantages that hash conflicts are completely avoided, and meanwhile, the linear time complexity based on the hash method is kept. Further, a parallel optimization scheme is proposed to accelerate data statistics. Experimental results show that the time efficiency and the space efficiency of the method on synthesis and real data sets are obviously superior to those of a baseline algorithm.

The invention provides a keyword private information retrieval method and device based on a hierarchical dynamic constant weight code, and the method comprises the following steps: constructing a multi-dimensional hypercube storage unit and an anti-collision hash function through a server, and mapping the coordinate of each data in the hypercube storage unit into a unique coordinate constant weight code in a hierarchical mapping manner; the client obtains a query vector and sends the query vector to the server; the server expands the query vector to obtain an expanded query vector, obtains a query result based on the expanded query vector, and returns the query result to the client; and the client decrypts the result and obtains the to-be-retrieved data according to the key value pair. According to the scheme, data mapping storage is carried out through the multi-dimensional hypercube storage unit and the anti-collision hash function, hash conflicts are reduced, global reconstruction is avoided, index updating time complexity is optimized through layered dynamic constant weight codes, and dynamic incremental expansion is achieved.

The application discloses a method, device and storage medium for solving IPFIX hash conflict, the method comprising receiving a data stream and enabling IPFIX function; obtaining a lookup key corresponding to the data stream and performing hash lookup, judging whether there is hash conflict according to the lookup result; when there is hash conflict, sending the lookup key to CPU, CPU calculating a new lookup key according to the lookup key and hash factor and performing hash lookup, judging whether there is hash conflict according to the lookup result; when there is hash conflict, continuing to calculate a new lookup key and performing lookup after adding one to the hash factor, until there is no hash conflict, CPU issuing a new ACL, which makes the matched data stream carry the hash factor; matching the data stream according to the new ACL, performing hash lookup according to the hash factor carried by the data stream and the lookup key corresponding to the data stream, and learning the data stream according to the lookup result. The application can solve the problem of IPFIX hash conflict.

The invention discloses an on-network aggregation method, device and system and a storage medium. The method comprises the following steps: S1, executing gradient aggregation logic by adopting a configurable matching table architecture; and S2, in the process of executing the gradient aggregation logic, Hash conflicts, resource overflow and floating point accumulation / sparse coding calculation are processed through an RISC-V calculation module. By adopting the technical scheme of the invention, a customized protocol and line speed processing specific to a data packet can be realized; and the abnormal condition which cannot be processed by the network card with limited network card flexibility can be processed.

The application discloses a data security union processing method and system, belongs to the technical field of data management, solves the problem that due to the inherent defect of hash algorithm, hash collision cannot be completely avoided, the introduction of cross-shard retrieval will destroy the shard parallelism, significantly increase the merging complexity, and lead to the problem that the shard number matching may cause the actual associated data to be scattered in different shards due to hash collision and cross-shard retrieval is needed. Including the following steps: dynamic sharding, according to the data heat of the primary key. Through the dynamic sharding, distributed bloom filter conflict pre-checking and local redirection query technology, combined with multi-thread parallel processing and real-time load balancing optimization, the proportion of cross-shard retrieval caused by hash collision is limited, the hash collision rate is reduced while the shard parallelism is ensured, the super large data set is merged, and the problem that the shard number matching may cause the actual associated data to be scattered in different shards due to hash collision and cross-shard retrieval is needed.

The invention discloses a data processing method and device for a hash table. The data processing method comprises the following steps: constructing a hash table; wherein the hash table comprises a first-level table and a second-level table, the first-level table comprises a plurality of hash buckets, each hash bucket comprises a plurality of first sub-slots, the second-level table comprises a plurality of address bars, the address bars are in one-to-one correspondence with the first sub-slots, and each address bar comprises a plurality of second sub-slots; obtaining a first key value pair, wherein the first key value pair comprises a first keyword and a first data item; combining data of preset bits in the first keyword to generate a first private key, and combining data of non-preset bits in the first keyword to generate a first signature value; selecting a target sub-slot from the idle first sub-slots; and storing the first signature value to the target sub-slot, and storing the first private key and the first data item to a second sub-slot of an address bar corresponding to the target sub-slot. The Hash conflict risk can be reduced, and the storage resource utilization rate and the access efficiency of the Hash table are improved.

The invention discloses a dynamic graph data storage method and device oriented to hybrid workloads. According to the method, aiming at the problem that various graph workloads are difficult to efficiently support in a single system in the prior art, a hash-based index construction, dynamic hierarchy edge storage design and a fine-grained concurrency control mechanism are provided. According to the method, efficient indexing of the vertexes is realized by adopting the hash table, and hash conflicts are solved in combination with a hash bucket strategy, so that the data query performance is improved. In the aspect of edge data storage, a compact array structure with a continuous memory is adopted, so that the traversal efficiency of graph data is optimized; meanwhile, aiming at the power law distribution characteristic of the graph data, the invention provides a dynamic hierarchical storage structure to improve the storage and updating efficiency, and an ordered storage strategy is adopted for the edge data of the vertex of the height number to efficiently support the graph pattern matching type workload. The invention provides a control mechanism, which improves the throughput of the system under a more novel working load while ensuring the data consistency.

The present application discloses a hash table processing method and apparatus, a hash table query method and apparatus, and a storage medium. The hash table processing method comprises: in response to a storage request for key-value pairs to be stored, determining a target hash bucket of the key-value pairs to be stored, so as to determine, on the basis of a storage state of the target hash bucket of the key-value pairs to be stored, whether there is a hash collision; when there is the hash collision, determining, on the basis of the location states of key-value pairs recorded by bitmaps and stored in the target hash bucket and / or the storage states of the hash buckets other than the target hash bucket, the storage locations of the key-value pairs to be stored; and storing, into the determined storage locations, the key-value pairs to be stored, using the bitmaps to record the location states of the key-value pairs, and optimizing the storage locations of the key-value pairs to be stored. In the present invention, by combining the bitmaps and the hash buckets, the storage resources of a hash table can be effectively used, and the storage locations of the key-value pairs are reasonably determined without expanding the hash buckets, thereby improving the utilization rate of the storage resources.

The invention discloses a switch dynamic load balancing method based on global traffic perception, which comprises the following steps of: constructing a global periodic traffic phase map to identify and synchronize a calculation phase and a synchronization phase of network traffic; before a synchronous phase arrives, a controller precomputes and issues a deterministic conflict-free routing strategy aiming at the phase, and a special phase forwarding table is formed in a switch; the switch atomically switches to the forwarding table at the start of the synchronization phase based on the global clock, performing deterministic path scheduling on the traffic. According to the method, passive response is replaced by active planning, the problems of perception lag and Hash conflict of a traditional load balancing algorithm are effectively relieved, transmission tail delay is reduced in combination with a micro scheduling mechanism, and the network communication efficiency and the computing power utilization rate of an artificial intelligence training cluster are improved.