42 results about "HashClash" patented technology

The present invention relates to an optimized access method and access system for a hash table in an OpenGauss hash connection. The method comprises the steps of constructing a hash table, completing the filling of a hash bucket array and a hash collision array; traversing the hash bucket array to detect whether a hash collision has occurred in the data stored at each location; and filling prompt information with the free space of the location information stored in the hash bucket array and the hash collision array based on the detection result. The method combines the advantages of the chain address method and the open addressing method, optimizes the implementation of the hash table from multiple perspectives, and distinguishes whether each conflicting data uses the chain address method or the open addressing method by filling in prompt information, thereby greatly improving the access performance of the hash table. In addition, the prompt information provided by the method can be used to determine whether a hash collision has occurred in the data without accessing the HashNext array, thereby improving the memory access and hash connection performance of the hash table.

The invention provides a character string coding and mapping method, system and device, a medium and a program product. The method comprises the steps that a to-be-coded character string is calculated through a self-defined coding polynomial to obtain a coding value corresponding to the character string; for each character string, allocating an index address to store a storage path corresponding to the character string in a storage space of a preset specification indicated by the index address, and storing the corresponding character string in a storage space indicated by the storage path; and storing the coded value of each character string and the corresponding index address in a lookup table by taking the coded value as an index so as to indicate a mapping relationship among the character string, the coded value, the index address and the storage path. The character string is coded through the polynomial, the prime number with the good hash characteristic serves as the polynomial cardinal number, the distribution uniformity of coded values is improved, and therefore the hash conflict probability is reduced; the realization of unique coding of the character string under the condition that the coding space is limited is ensured, and the system stability and the retrieval accuracy are improved.

The application discloses a unique hash serial number generation method and system, solves the problem of a large hash value range interval, and improves the space utilization rate and data query efficiency of an object with a string as an index during storage. The technical scheme is as follows: a string is processed to generate a hash serial number which is self-incremented from 0, the hash serial number is used as an index to replace the original storage data structure which needs a string as an index; a cblock_map data cache structure is used, the cblock_map uses a hash value of a hash item as an index when storing the hash item, and contains information such as a string and a hash serial number. In the running process of the method, the cblock_map memory structure is stable, all possible spaces are prepared in advance in the construction method, and there is no scenario of recreating an object and copying data after the space is full. The cblock_map provides efficient and stable read and write capabilities, supports O(1) efficiency of the index storage position of the hash value, and even in the case of hash conflict, the query time can be controlled in a constant range through the way of list traversal.

According to the updating method of the FC switch monitoring lookup table based on the LFU, the storage of the monitoring port lookup table is realized by adopting the idea of Robin Hood Hashing, and according to the strategy, a key with a longer detection distance hijacks the position of a key with a shorter detection distance when a Hash conflict occurs during insertion, so that the query performance is optimized, and the updating method is suitable for a high-performance network scene; secondly, modular design is adopted, message IDs existing in a monitoring port lookup table are stored in a grouped mode according to the access frequency through a dictionary and a double-linked list structure, and global sorting is avoided; thirdly, when the message ID stored in the monitoring port query table is full and updated, a double-layer elimination strategy is adopted, the message ID which is least recently used can be efficiently positioned, all frequencies do not need to be traversed, rapid updating is achieved, the method is suitable for a high-concurrency or large-scale FC network communication scene, and therefore the judgment efficiency of the target motion state is improved.

This invention discloses a DDoS attack detection method based on heteroscedastic unscented Kalman filtering. Addressing the issues of limited resources in edge networks and low anomaly detection accuracy and high latency in low signal-to-noise ratio environments, this invention utilizes Sketch for full-scale, unsampled traffic feature extraction in the programmable data plane and reports observations to the control plane through differential processing. In the control plane, a third-order state-space model incorporating instantaneous traffic rate, rate of change, and acceleration is innovatively constructed. An additive heteroscedastic noise model is proposed to adaptively handle Sketch hash collisions and traffic shot noise, and unscented Kalman filtering is employed for state estimation. Simultaneously, an intelligent hybrid control mechanism is introduced to overcome filtering lag, and finally, NIS and CUSUM are combined for dual-modal anomaly detection, outputting the system state. This invention enables high-precision real-time detection of DDoS attacks with extremely low resource overhead, effectively solving the trade-off between measurement and detection in existing technologies.

The invention provides a data transmission method, device, equipment and product, and relates to the technical field of communication networks. The method comprises: receiving a first data packet, the first data packet being a current to-be-transmitted data packet in a logic frame, the first data packet comprising first equalization sequence information; determining a value and a first Hash conflict number corresponding to the first balancing sequence information, the value being used for indicating transmission of the first data packet according to a load balancing mode; and releasing the corresponding value when a release condition is satisfied, the release condition being associated with the number of first hash conflicts and whether the first data packet is a tail packet of the logic frame. According to the mode, the storage space overhead of the value is reduced.

The invention provides a data storage method, device and equipment, and the method comprises the steps: determining a target bit number P for newly-added to-be-stored data in a bit cutting hash data structure; the bit cutting hash data structure comprises a plurality of hash buckets, and the target bit number P is used for enabling the to-be-stored data and the stored data in the bit cutting hash data structure not to have hash conflicts; determining a BSS value of each bit based on the stored data and the to-be-stored data, and determining P feature bits based on the BSS value of each bit; based on the corresponding numerical values of the P feature bits in the to-be-stored data, determining a corresponding target hash bucket of the to-be-stored data in the bit cutting hash data structure; and storing the to-be-stored data in the target hash bucket. According to the scheme, the accurate matching performance of the data can be improved.

The invention provides a method and device for identifying duplicated data based on hierarchical hash lookup, and the method comprises the steps: executing two independent hash operations in parallel for input data to obtain two lookup addresses, and constructing a lookup structure consisting of a plurality of levels of mutually independent hash tables and a residual table. The method comprises the following steps: firstly, carrying out double-hash search on data in a first-level hash table, and cascading the data to a next-level hash table if a conflict occurs until all hierarchical tables are accessed; and for the data which still conflict, sequentially searching in the residual table. According to the method, the probability of Hash conflicts is effectively reduced by adopting the multi-stage independent Hash table and a double-Hash lookup mechanism; more importantly, most of the search delays are limited in a predictable limited clock period by the hierarchical pipeline design, so that the search delay under the worst condition is obviously optimized, and an extremely high system throughput rate is realized in combination with multi-pass parallel.

The application discloses a kind of spaceborne high-speed hybrid hash routing lookup method and system.For improving routing lookup efficiency, pre-processing is set in the front end of core routing lookup module, and multi-channel IP message data stream arbitration is combined into two ways, routing lookup is carried out simultaneously in two ways, and finally the result of routing lookup is distributed to corresponding channel;Secondly, the hybrid hash routing lookup mode of routing lookup process is combined with segmented parallel lookup, multiple hash bucket parallel lookup and multiple hash algorithm lookup, and each result is filtered and output according to table item priority after obtaining the query result.The application combines the characteristics of hash table lookup time complexity O (1), has good query performance, effectively reduces the probability of hash conflict, greatly improves the probability of successful storage of hash table item on the basis of saving hardware storage resources, greatly improves the query accuracy, and solves the demand of high-speed IP routing lookup on satellite product.

The invention provides a hash conflict processing method and device, equipment, a storage medium and a product. The method comprises the following steps: acquiring a conflict element and an original Hash slot of the conflict element in a Hash table, and detecting a Hash slot in an adjacent window on the rear side of the original Hash slot backwards by taking the original Hash slot as a starting point; if the empty Hash slot does not exist in the adjacent window on the rear side, moving elements stored in the Hash slot in the Hash table to enable the empty Hash slot to exist in the adjacent window on the rear side, and inserting conflict elements into the empty Hash slot; configuring a preset identifier for the original Hash slot and the empty Hash slot into which the conflict element is inserted, and generating an adjacent bitmap of the original Hash slot according to the configured preset identifier; and when a conflict element search instruction is received, reading the adjacent bitmap, determining a to-be-matched hash slot configured with a preset identifier, and determining the to-be-matched hash slot of which the stored element is the same as the conflict element as a target hash slot. According to the method, the cache hit rate and the search efficiency can be improved when the Hash conflict is processed.

The invention discloses a distributed model training on-network convergence method based on a programmable switch, which comprises the following steps of: calculating a register slot index position when a model gradient package is converged on the programmable switch by adopting a plurality of hash functions; the problem of Hash collision is solved by dividing the available memory of the programmable switch into a plurality of Hash intervals, and compared with the existing mode of only using a single Hash function, the proportion of Hash collision during convergence of the model gradient packets is effectively reduced, the number of the model gradient packets converged on the network on the programmable switch is increased, and the convergence efficiency of the model gradient packets is improved. Therefore, the model gradient flow sent by the programmable switch to the model convergence server and the competition of the model gradient flow on the limited bandwidth of the inlet of the model convergence server are reduced, and the purposes of accelerating distributed model training and reducing the network flow load of the data center are achieved.

This invention discloses a method and system for local differential privacy aggregate value data aggregation based on sketches and sampling, belonging to the field of privacy computing and big data analysis technology. The method includes: initializing and disclosing sketch structure parameters on the server side; constructing aggregate value data on the user side, randomly selecting counter indices and determining the hash collision set; randomizing the hash collision set elements on the user side according to the aggregation function type, calculating the hit rate and performing adaptive pruning; reporting triples on the user side; and accumulating and updating the sketch counter on the server side, calculating the aggregated statistics through median estimation. This invention replaces traditional multi-element perturbation with sketch sampling and lightweight reporting, without requiring additional privacy budget consumption, effectively solving the noise accumulation problem, significantly reducing computational and communication overhead, and achieving a balance between privacy protection and statistical utility.

The invention provides a message classification method and device, and the method comprises the steps: taking all preset quintuples as a statistical sample, carrying out the calculation, and obtaining the information entropy of each quintuple byte, and enabling the preset quintuples to be quintuples configured by network nodes; a first Hash result corresponding to a preset quintuple is obtained through calculation according to a preset Hash algorithm, first configuration information is stored with the first Hash result as an address, the Hash result comprises two necessary bytes, each necessary byte is obtained through bitwise exclusive or of a plurality of quintuple bytes, and the first configuration information is stored according to the bitwise exclusive or of the quintuple bytes. Two quintuple bytes with the maximum information entropy participate in operation of two necessary bytes respectively; a second hash result corresponding to an actual quintuple is obtained through calculation according to a preset hash algorithm, second configuration information is inquired with the second hash result as an address, and the actual quintuple is a quintuple carried by the message; and if the second configuration information is queried, carrying out message classification according to the second configuration information. By means of the message classification method and device, the hash conflict situation during message classification can be reduced.

A network device includes a storage device, a hardware-accelerated forwarding circuit, and a network processing unit (NPU). The storage device stores a software flow table, where the software flow table includes a plurality of table entries. The hardware-accelerated forwarding circuit receives a first packet from a network, obtains a first hash value of the first packet, and determines that the first hash value encounters a hash collision in a hardware flow table. The NPU receives the first packet from the hardware-accelerated forwarding circuit, and deals with forwarding of the first packet according to forwarding information recorded in a table entry when the first packet hits the table entry of the software flow table.

Methods and systems for estimating the frequency table of a data set stored in a probabilistic data structure such as a counting Bloom filter or Count-Min sketch. The system observes the distribution of values in the counters of the data structure (e.g., how many counters have value 0, 1, 2, etc.) Using these observed values, the system iteratively calculates the number of unique items in the data set that appear with specific frequencies. The method accounts for hash collisions by modeling the probability that a counter value results from a single item versus multiple colliding items, allowing for the reconstruction of the item frequency distribution without querying individual elements.

The invention discloses a data deduplication method and system applied to automatic acquisition of quality information, belongs to the technical field of data processing, and aims to solve the technical problem of how to overcome hash conflicts and data sensitivity in data deduplication. Comprising the following steps: selecting a to-be-crawled website and a theme module based on a predefined webpage crawling duplicate removal rule; periodically carrying out website data crawling, and recording the website data crawled by each website into a website library table corresponding to the website; performing in-site data deduplication operation on the website data under each website based on the title and the release time, performing inter-site data deduplication operation on the website data among different websites based on the hash value of the website data, and writing the website data of each website after the inter-site data deduplication operation and the hash value of the website data into a temporary library; and performing data deduplication operation on the website data in the temporary library based on the hash value of the website data, and writing the deduplicated website data and the hash value of the website data into a formal library.

The application discloses a method, device and storage medium for solving IPFIX hash conflict, the method comprising receiving a data stream and enabling IPFIX function; obtaining a lookup key corresponding to the data stream and performing hash lookup, judging whether there is hash conflict according to the lookup result; when there is hash conflict, sending the lookup key to CPU, CPU calculating a new lookup key according to the lookup key and hash factor and performing hash lookup, judging whether there is hash conflict according to the lookup result; when there is hash conflict, continuing to calculate a new lookup key and performing lookup after adding one to the hash factor, until there is no hash conflict, CPU issuing a new ACL, which makes the matched data stream carry the hash factor; matching the data stream according to the new ACL, performing hash lookup according to the hash factor carried by the data stream and the lookup key corresponding to the data stream, and learning the data stream according to the lookup result. The application can solve the problem of IPFIX hash conflict.

The invention discloses an on-network aggregation method, device and system and a storage medium. The method comprises the following steps: S1, executing gradient aggregation logic by adopting a configurable matching table architecture; and S2, in the process of executing the gradient aggregation logic, Hash conflicts, resource overflow and floating point accumulation / sparse coding calculation are processed through an RISC-V calculation module. By adopting the technical scheme of the invention, a customized protocol and line speed processing specific to a data packet can be realized; and the abnormal condition which cannot be processed by the network card with limited network card flexibility can be processed.

The application discloses a data security union processing method and system, belongs to the technical field of data management, solves the problem that due to the inherent defect of hash algorithm, hash collision cannot be completely avoided, the introduction of cross-shard retrieval will destroy the shard parallelism, significantly increase the merging complexity, and lead to the problem that the shard number matching may cause the actual associated data to be scattered in different shards due to hash collision and cross-shard retrieval is needed. Including the following steps: dynamic sharding, according to the data heat of the primary key. Through the dynamic sharding, distributed bloom filter conflict pre-checking and local redirection query technology, combined with multi-thread parallel processing and real-time load balancing optimization, the proportion of cross-shard retrieval caused by hash collision is limited, the hash collision rate is reduced while the shard parallelism is ensured, the super large data set is merged, and the problem that the shard number matching may cause the actual associated data to be scattered in different shards due to hash collision and cross-shard retrieval is needed.

The invention discloses a data processing method and device for a hash table. The data processing method comprises the following steps: constructing a hash table; wherein the hash table comprises a first-level table and a second-level table, the first-level table comprises a plurality of hash buckets, each hash bucket comprises a plurality of first sub-slots, the second-level table comprises a plurality of address bars, the address bars are in one-to-one correspondence with the first sub-slots, and each address bar comprises a plurality of second sub-slots; obtaining a first key value pair, wherein the first key value pair comprises a first keyword and a first data item; combining data of preset bits in the first keyword to generate a first private key, and combining data of non-preset bits in the first keyword to generate a first signature value; selecting a target sub-slot from the idle first sub-slots; and storing the first signature value to the target sub-slot, and storing the first private key and the first data item to a second sub-slot of an address bar corresponding to the target sub-slot. The Hash conflict risk can be reduced, and the storage resource utilization rate and the access efficiency of the Hash table are improved.

The invention discloses a dynamic graph data storage method and device oriented to hybrid workloads. According to the method, aiming at the problem that various graph workloads are difficult to efficiently support in a single system in the prior art, a hash-based index construction, dynamic hierarchy edge storage design and a fine-grained concurrency control mechanism are provided. According to the method, efficient indexing of the vertexes is realized by adopting the hash table, and hash conflicts are solved in combination with a hash bucket strategy, so that the data query performance is improved. In the aspect of edge data storage, a compact array structure with a continuous memory is adopted, so that the traversal efficiency of graph data is optimized; meanwhile, aiming at the power law distribution characteristic of the graph data, the invention provides a dynamic hierarchical storage structure to improve the storage and updating efficiency, and an ordered storage strategy is adopted for the edge data of the vertex of the height number to efficiently support the graph pattern matching type workload. The invention provides a control mechanism, which improves the throughput of the system under a more novel working load while ensuring the data consistency.

The present application discloses a hash table processing method and apparatus, a hash table query method and apparatus, and a storage medium. The hash table processing method comprises: in response to a storage request for key-value pairs to be stored, determining a target hash bucket of the key-value pairs to be stored, so as to determine, on the basis of a storage state of the target hash bucket of the key-value pairs to be stored, whether there is a hash collision; when there is the hash collision, determining, on the basis of the location states of key-value pairs recorded by bitmaps and stored in the target hash bucket and / or the storage states of the hash buckets other than the target hash bucket, the storage locations of the key-value pairs to be stored; and storing, into the determined storage locations, the key-value pairs to be stored, using the bitmaps to record the location states of the key-value pairs, and optimizing the storage locations of the key-value pairs to be stored. In the present invention, by combining the bitmaps and the hash buckets, the storage resources of a hash table can be effectively used, and the storage locations of the key-value pairs are reasonably determined without expanding the hash buckets, thereby improving the utilization rate of the storage resources.

The invention discloses a switch dynamic load balancing method based on global traffic perception, which comprises the following steps of: constructing a global periodic traffic phase map to identify and synchronize a calculation phase and a synchronization phase of network traffic; before a synchronous phase arrives, a controller precomputes and issues a deterministic conflict-free routing strategy aiming at the phase, and a special phase forwarding table is formed in a switch; the switch atomically switches to the forwarding table at the start of the synchronization phase based on the global clock, performing deterministic path scheduling on the traffic. According to the method, passive response is replaced by active planning, the problems of perception lag and Hash conflict of a traditional load balancing algorithm are effectively relieved, transmission tail delay is reduced in combination with a micro scheduling mechanism, and the network communication efficiency and the computing power utilization rate of an artificial intelligence training cluster are improved.

The embodiment of the invention provides a flow processing system and method, a data processing method and device, a medium and a product. Wherein the data sending end foreknows a Hash scheduling rule and an input field (namely, a source port) adopted by the switch, and reversely constructs a source port number meeting the Hash scheduling rule of the switch according to an identifier of a physical link sending target data, so as to ensure that a Hash result of the source port number hits a specific link, connected with the switch, of a target end. Sufficient entropy values are introduced through active modulation of source port numbers, so that Hash conflicts can be effectively relieved; a data transmitting end implements an effective load balancing strategy on an uplink, and the transmitting end reversely constructs a source port number to ensure that different data streams are forwarded by a specific downlink, so that the flow distribution of the downlink and the uplink keep a corresponding relationship, and the flow in the downlink direction also realizes balanced distribution; and the problem of utilization rate imbalance of the physical link can be effectively relieved.

The invention provides a data stream hash conflict solving method and device and a readable storage medium. The method comprises the following steps: receiving data streams of a plurality of external devices; performing hash calculation on each data stream to obtain a hash value of each data stream; if the hash value of the data stream appears for the first time, determining the data stream as a normal data stream, directing the normal data stream to an IPFIX engine, and processing the normal data stream by using the IPFIX engine; and if the hash value of the data stream is the same as the hash value of one of the normal data streams, determining the data stream as a conflict data stream, redirecting each conflict data stream to an ACL engine, and processing each conflict data stream by using the ACL engine. According to the method, the ACL engine is used for processing the conflicting data streams, the normal data streams are processed by the IPFIX engine, hardware and software collaboration is achieved, the conflicting data streams and the normal data streams are correctly counted, loss of the conflicting data streams in the IPFIX engine is reduced, and monitoring accuracy is guaranteed.

The invention provides a perfect hash quick searching method for an embedded system, and relates to the technical field of hash searching, the method comprises a construction stage and a searching stage, the construction stage comprises the following steps: mapping a static key set into a plurality of buckets through a hash function, and arranging all the buckets in a descending order according to the size; distributing a unique guide value for each bucket, generating a conflict-free hash position and marking an occupation state; establishing an original index of a reverse lookup table record key; the searching stage comprises the following steps: calculating an initial hash value of an input key and determining a bucket to which the initial hash value is mapped; acquiring a guidance value of the determined bucket and calculating the position of the bucket; and returning a real index value corresponding to the position through the reverse lookup table. The method provided by the invention not only can eliminate Hash conflicts during operation and guarantee real-time performance, but also can well adapt to embedded resource limitation, and in addition, can effectively improve system reliability. According to the method, the most sensitive real-time performance and resource bottleneck problems of an embedded scene can be fundamentally avoided.

The invention discloses a Hash conflict processing method and device and a storage medium, and relates to the technical field of communication, and the Hash conflict processing method comprises the steps that in response to the situation that the data size of to-be-inserted data is larger than the idle capacity of a first target sub-table in extensible Hash, whether the first target sub-table reaches the sub-table splitting upper limit or not is determined, the first target sub-table is a sub-table in the extensible hash indicated by the hash value of the to-be-inserted data; under the condition that the first target sub-expression reaches the sub-table splitting upper limit, expanding the first target sub-table to obtain an expanded sub-table; and inserting the to-be-inserted data into the expansion sub-table. According to the invention, the Hash conflict is effectively solved, and the stability of the system is guaranteed.

The invention discloses an extensible compression storage method and system for a real-time database. The method comprises the following steps: dividing data in a real-time database table into a fixed domain and a dynamic domain, and adopting a separated metadata management and data storage architecture; metadata information and data of the fixed domain are stored in a record line form; an extensible space is reserved for metadata information of the dynamic domain, data of the metadata information is stored in a key value pair format, and each key value pair comprises a recording primary key, a dynamic domain number and a dynamic domain value; positioning a storage position of a dynamic domain key value pair based on a Hash mapping mechanism for recording a primary key, and processing Hash conflicts by adopting two double linked lists, namely a primary linked list and a secondary linked list; default value attributes are preset for each dynamic domain, so that compressed storage of sparse dynamic domain data is achieved; a unified data access interface is provided, and the bottom storage difference between the fixed domain and the dynamic domain is shielded. According to the method, the problems of poor points in the aspects of online expansion, storage efficiency, access performance, use convenience and the like of the real-time database are solved.

The application relates to a device communication method and device based on a single bus communication system and a quick shutdown system, and relates to the technical field of device communication. The method embodiment comprises the following steps: obtaining device numbers of a plurality of slave devices, taking the device numbers as keys and taking feature data fields in the device numbers as key values to construct a hash table; sending a request by a master device to a plurality of target slave devices with the same key values, so that the plurality of target slave devices send reply data comprising target device numbers to the master device in turn based on an exponential backoff algorithm, the master device performs hash calculation on the target device numbers based on the received reply data to obtain updated key values, and the hash table is updated; and the master device communicates with the corresponding slave devices based on the updated hash table. The embodiment avoids the problem of hash conflict, simplifies wiring and reduces costs, and effectively guarantees the data transmission accuracy and system operation stability in the communication process of the master and slave devices.

Embodiments of the present application provide a flow processing system and method, a data processing method, equipment, a medium and a product. In the embodiments, a data sending end knows a hash scheduling rule adopted by a switch and an input field (i.e., a source port), and according to an identifier of a physical link from which target data is sent, a source port number satisfying the hash scheduling rule of the switch is reversely constructed, and it is ensured that a hash result of the source port number hits a specific link connected to the switch by a destination end. By actively modulating the source port number to introduce sufficient entropy, hash conflicts can be effectively alleviated. The data sending end implements an effective load balancing strategy on an uplink, and the sending end reversely constructs the source port number to ensure that different data streams are forwarded by specific downlinks, so that the flow distribution of the downlinks corresponds to the uplink, and the flow in the downlink direction is also balanced, which can effectively alleviate the imbalance of the utilization rate of the physical link.