27 results about "HashClash" patented technology

The application discloses a unique hash serial number generation method and system, solves the problem of a large hash value range interval, and improves the space utilization rate and data query efficiency of an object with a string as an index during storage. The technical scheme is as follows: a string is processed to generate a hash serial number which is self-incremented from 0, the hash serial number is used as an index to replace the original storage data structure which needs a string as an index; a cblock_map data cache structure is used, the cblock_map uses a hash value of a hash item as an index when storing the hash item, and contains information such as a string and a hash serial number. In the running process of the method, the cblock_map memory structure is stable, all possible spaces are prepared in advance in the construction method, and there is no scenario of recreating an object and copying data after the space is full. The cblock_map provides efficient and stable read and write capabilities, supports O(1) efficiency of the index storage position of the hash value, and even in the case of hash conflict, the query time can be controlled in a constant range through the way of list traversal.

According to the updating method of the FC switch monitoring lookup table based on the LFU, the storage of the monitoring port lookup table is realized by adopting the idea of Robin Hood Hashing, and according to the strategy, a key with a longer detection distance hijacks the position of a key with a shorter detection distance when a Hash conflict occurs during insertion, so that the query performance is optimized, and the updating method is suitable for a high-performance network scene; secondly, modular design is adopted, message IDs existing in a monitoring port lookup table are stored in a grouped mode according to the access frequency through a dictionary and a double-linked list structure, and global sorting is avoided; thirdly, when the message ID stored in the monitoring port query table is full and updated, a double-layer elimination strategy is adopted, the message ID which is least recently used can be efficiently positioned, all frequencies do not need to be traversed, rapid updating is achieved, the method is suitable for a high-concurrency or large-scale FC network communication scene, and therefore the judgment efficiency of the target motion state is improved.

This invention discloses a DDoS attack detection method based on heteroscedastic unscented Kalman filtering. Addressing the issues of limited resources in edge networks and low anomaly detection accuracy and high latency in low signal-to-noise ratio environments, this invention utilizes Sketch for full-scale, unsampled traffic feature extraction in the programmable data plane and reports observations to the control plane through differential processing. In the control plane, a third-order state-space model incorporating instantaneous traffic rate, rate of change, and acceleration is innovatively constructed. An additive heteroscedastic noise model is proposed to adaptively handle Sketch hash collisions and traffic shot noise, and unscented Kalman filtering is employed for state estimation. Simultaneously, an intelligent hybrid control mechanism is introduced to overcome filtering lag, and finally, NIS and CUSUM are combined for dual-modal anomaly detection, outputting the system state. This invention enables high-precision real-time detection of DDoS attacks with extremely low resource overhead, effectively solving the trade-off between measurement and detection in existing technologies.

The invention provides a method and device for identifying duplicated data based on hierarchical hash lookup, and the method comprises the steps: executing two independent hash operations in parallel for input data to obtain two lookup addresses, and constructing a lookup structure consisting of a plurality of levels of mutually independent hash tables and a residual table. The method comprises the following steps: firstly, carrying out double-hash search on data in a first-level hash table, and cascading the data to a next-level hash table if a conflict occurs until all hierarchical tables are accessed; and for the data which still conflict, sequentially searching in the residual table. According to the method, the probability of Hash conflicts is effectively reduced by adopting the multi-stage independent Hash table and a double-Hash lookup mechanism; more importantly, most of the search delays are limited in a predictable limited clock period by the hierarchical pipeline design, so that the search delay under the worst condition is obviously optimized, and an extremely high system throughput rate is realized in combination with multi-pass parallel.

The invention provides a hash conflict processing method and device, equipment, a storage medium and a product. The method comprises the following steps: acquiring a conflict element and an original Hash slot of the conflict element in a Hash table, and detecting a Hash slot in an adjacent window on the rear side of the original Hash slot backwards by taking the original Hash slot as a starting point; if the empty Hash slot does not exist in the adjacent window on the rear side, moving elements stored in the Hash slot in the Hash table to enable the empty Hash slot to exist in the adjacent window on the rear side, and inserting conflict elements into the empty Hash slot; configuring a preset identifier for the original Hash slot and the empty Hash slot into which the conflict element is inserted, and generating an adjacent bitmap of the original Hash slot according to the configured preset identifier; and when a conflict element search instruction is received, reading the adjacent bitmap, determining a to-be-matched hash slot configured with a preset identifier, and determining the to-be-matched hash slot of which the stored element is the same as the conflict element as a target hash slot. According to the method, the cache hit rate and the search efficiency can be improved when the Hash conflict is processed.

This invention discloses a method and system for local differential privacy aggregate value data aggregation based on sketches and sampling, belonging to the field of privacy computing and big data analysis technology. The method includes: initializing and disclosing sketch structure parameters on the server side; constructing aggregate value data on the user side, randomly selecting counter indices and determining the hash collision set; randomizing the hash collision set elements on the user side according to the aggregation function type, calculating the hit rate and performing adaptive pruning; reporting triples on the user side; and accumulating and updating the sketch counter on the server side, calculating the aggregated statistics through median estimation. This invention replaces traditional multi-element perturbation with sketch sampling and lightweight reporting, without requiring additional privacy budget consumption, effectively solving the noise accumulation problem, significantly reducing computational and communication overhead, and achieving a balance between privacy protection and statistical utility.

A network device includes a storage device, a hardware-accelerated forwarding circuit, and a network processing unit (NPU). The storage device stores a software flow table, where the software flow table includes a plurality of table entries. The hardware-accelerated forwarding circuit receives a first packet from a network, obtains a first hash value of the first packet, and determines that the first hash value encounters a hash collision in a hardware flow table. The NPU receives the first packet from the hardware-accelerated forwarding circuit, and deals with forwarding of the first packet according to forwarding information recorded in a table entry when the first packet hits the table entry of the software flow table.

Methods and systems for estimating the frequency table of a data set stored in a probabilistic data structure such as a counting Bloom filter or Count-Min sketch. The system observes the distribution of values in the counters of the data structure (e.g., how many counters have value 0, 1, 2, etc.) Using these observed values, the system iteratively calculates the number of unique items in the data set that appear with specific frequencies. The method accounts for hash collisions by modeling the probability that a counter value results from a single item versus multiple colliding items, allowing for the reconstruction of the item frequency distribution without querying individual elements.

The application discloses a method, device and storage medium for solving IPFIX hash conflict, the method comprising receiving a data stream and enabling IPFIX function; obtaining a lookup key corresponding to the data stream and performing hash lookup, judging whether there is hash conflict according to the lookup result; when there is hash conflict, sending the lookup key to CPU, CPU calculating a new lookup key according to the lookup key and hash factor and performing hash lookup, judging whether there is hash conflict according to the lookup result; when there is hash conflict, continuing to calculate a new lookup key and performing lookup after adding one to the hash factor, until there is no hash conflict, CPU issuing a new ACL, which makes the matched data stream carry the hash factor; matching the data stream according to the new ACL, performing hash lookup according to the hash factor carried by the data stream and the lookup key corresponding to the data stream, and learning the data stream according to the lookup result. The application can solve the problem of IPFIX hash conflict.

The invention discloses an on-network aggregation method, device and system and a storage medium. The method comprises the following steps: S1, executing gradient aggregation logic by adopting a configurable matching table architecture; and S2, in the process of executing the gradient aggregation logic, Hash conflicts, resource overflow and floating point accumulation / sparse coding calculation are processed through an RISC-V calculation module. By adopting the technical scheme of the invention, a customized protocol and line speed processing specific to a data packet can be realized; and the abnormal condition which cannot be processed by the network card with limited network card flexibility can be processed.

The application discloses a data security union processing method and system, belongs to the technical field of data management, solves the problem that due to the inherent defect of hash algorithm, hash collision cannot be completely avoided, the introduction of cross-shard retrieval will destroy the shard parallelism, significantly increase the merging complexity, and lead to the problem that the shard number matching may cause the actual associated data to be scattered in different shards due to hash collision and cross-shard retrieval is needed. Including the following steps: dynamic sharding, according to the data heat of the primary key. Through the dynamic sharding, distributed bloom filter conflict pre-checking and local redirection query technology, combined with multi-thread parallel processing and real-time load balancing optimization, the proportion of cross-shard retrieval caused by hash collision is limited, the hash collision rate is reduced while the shard parallelism is ensured, the super large data set is merged, and the problem that the shard number matching may cause the actual associated data to be scattered in different shards due to hash collision and cross-shard retrieval is needed.

The present application discloses a hash table processing method and apparatus, a hash table query method and apparatus, and a storage medium. The hash table processing method comprises: in response to a storage request for key-value pairs to be stored, determining a target hash bucket of the key-value pairs to be stored, so as to determine, on the basis of a storage state of the target hash bucket of the key-value pairs to be stored, whether there is a hash collision; when there is the hash collision, determining, on the basis of the location states of key-value pairs recorded by bitmaps and stored in the target hash bucket and / or the storage states of the hash buckets other than the target hash bucket, the storage locations of the key-value pairs to be stored; and storing, into the determined storage locations, the key-value pairs to be stored, using the bitmaps to record the location states of the key-value pairs, and optimizing the storage locations of the key-value pairs to be stored. In the present invention, by combining the bitmaps and the hash buckets, the storage resources of a hash table can be effectively used, and the storage locations of the key-value pairs are reasonably determined without expanding the hash buckets, thereby improving the utilization rate of the storage resources.

The invention discloses a switch dynamic load balancing method based on global traffic perception, which comprises the following steps of: constructing a global periodic traffic phase map to identify and synchronize a calculation phase and a synchronization phase of network traffic; before a synchronous phase arrives, a controller precomputes and issues a deterministic conflict-free routing strategy aiming at the phase, and a special phase forwarding table is formed in a switch; the switch atomically switches to the forwarding table at the start of the synchronization phase based on the global clock, performing deterministic path scheduling on the traffic. According to the method, passive response is replaced by active planning, the problems of perception lag and Hash conflict of a traditional load balancing algorithm are effectively relieved, transmission tail delay is reduced in combination with a micro scheduling mechanism, and the network communication efficiency and the computing power utilization rate of an artificial intelligence training cluster are improved.

The embodiment of the invention provides a flow processing system and method, a data processing method and device, a medium and a product. Wherein the data sending end foreknows a Hash scheduling rule and an input field (namely, a source port) adopted by the switch, and reversely constructs a source port number meeting the Hash scheduling rule of the switch according to an identifier of a physical link sending target data, so as to ensure that a Hash result of the source port number hits a specific link, connected with the switch, of a target end. Sufficient entropy values are introduced through active modulation of source port numbers, so that Hash conflicts can be effectively relieved; a data transmitting end implements an effective load balancing strategy on an uplink, and the transmitting end reversely constructs a source port number to ensure that different data streams are forwarded by a specific downlink, so that the flow distribution of the downlink and the uplink keep a corresponding relationship, and the flow in the downlink direction also realizes balanced distribution; and the problem of utilization rate imbalance of the physical link can be effectively relieved.

The invention provides a data stream hash conflict solving method and device and a readable storage medium. The method comprises the following steps: receiving data streams of a plurality of external devices; performing hash calculation on each data stream to obtain a hash value of each data stream; if the hash value of the data stream appears for the first time, determining the data stream as a normal data stream, directing the normal data stream to an IPFIX engine, and processing the normal data stream by using the IPFIX engine; and if the hash value of the data stream is the same as the hash value of one of the normal data streams, determining the data stream as a conflict data stream, redirecting each conflict data stream to an ACL engine, and processing each conflict data stream by using the ACL engine. According to the method, the ACL engine is used for processing the conflicting data streams, the normal data streams are processed by the IPFIX engine, hardware and software collaboration is achieved, the conflicting data streams and the normal data streams are correctly counted, loss of the conflicting data streams in the IPFIX engine is reduced, and monitoring accuracy is guaranteed.

The invention discloses a Hash conflict processing method and device and a storage medium, and relates to the technical field of communication, and the Hash conflict processing method comprises the steps that in response to the situation that the data size of to-be-inserted data is larger than the idle capacity of a first target sub-table in extensible Hash, whether the first target sub-table reaches the sub-table splitting upper limit or not is determined, the first target sub-table is a sub-table in the extensible hash indicated by the hash value of the to-be-inserted data; under the condition that the first target sub-expression reaches the sub-table splitting upper limit, expanding the first target sub-table to obtain an expanded sub-table; and inserting the to-be-inserted data into the expansion sub-table. According to the invention, the Hash conflict is effectively solved, and the stability of the system is guaranteed.

The invention discloses an extensible compression storage method and system for a real-time database. The method comprises the following steps: dividing data in a real-time database table into a fixed domain and a dynamic domain, and adopting a separated metadata management and data storage architecture; metadata information and data of the fixed domain are stored in a record line form; an extensible space is reserved for metadata information of the dynamic domain, data of the metadata information is stored in a key value pair format, and each key value pair comprises a recording primary key, a dynamic domain number and a dynamic domain value; positioning a storage position of a dynamic domain key value pair based on a Hash mapping mechanism for recording a primary key, and processing Hash conflicts by adopting two double linked lists, namely a primary linked list and a secondary linked list; default value attributes are preset for each dynamic domain, so that compressed storage of sparse dynamic domain data is achieved; a unified data access interface is provided, and the bottom storage difference between the fixed domain and the dynamic domain is shielded. According to the method, the problems of poor points in the aspects of online expansion, storage efficiency, access performance, use convenience and the like of the real-time database are solved.

The application relates to a device communication method and device based on a single bus communication system and a quick shutdown system, and relates to the technical field of device communication. The method embodiment comprises the following steps: obtaining device numbers of a plurality of slave devices, taking the device numbers as keys and taking feature data fields in the device numbers as key values to construct a hash table; sending a request by a master device to a plurality of target slave devices with the same key values, so that the plurality of target slave devices send reply data comprising target device numbers to the master device in turn based on an exponential backoff algorithm, the master device performs hash calculation on the target device numbers based on the received reply data to obtain updated key values, and the hash table is updated; and the master device communicates with the corresponding slave devices based on the updated hash table. The embodiment avoids the problem of hash conflict, simplifies wiring and reduces costs, and effectively guarantees the data transmission accuracy and system operation stability in the communication process of the master and slave devices.

Embodiments of the present application provide a flow processing system and method, a data processing method, equipment, a medium and a product. In the embodiments, a data sending end knows a hash scheduling rule adopted by a switch and an input field (i.e., a source port), and according to an identifier of a physical link from which target data is sent, a source port number satisfying the hash scheduling rule of the switch is reversely constructed, and it is ensured that a hash result of the source port number hits a specific link connected to the switch by a destination end. By actively modulating the source port number to introduce sufficient entropy, hash conflicts can be effectively alleviated. The data sending end implements an effective load balancing strategy on an uplink, and the sending end reversely constructs the source port number to ensure that different data streams are forwarded by specific downlinks, so that the flow distribution of the downlinks corresponds to the uplink, and the flow in the downlink direction is also balanced, which can effectively alleviate the imbalance of the utilization rate of the physical link.

In the field of optical communication technologies, an artificial intelligence computing system and method are provided to resolve a problem of low communication efficiency caused by a hash conflict and traffic imbalance. Disclosed embodiments provide a link communication between computing nodes in a process of implementing an AI data computing task is implemented through an optical switching network. Each computing node supports control of cross links between input ports and output ports of an optical switch switching assembly in the optical switching network. In a process in which a computing node performs a computing task, a control unit of the computing node performs link switching based on a requirement of the computing node.

The invention provides a data processing method, and the method is applied to an artificial intelligence (AI) processor which supports a single instruction multiple data (SIMD), and the method is applied to the artificial intelligence (AI) processor which supports the SIMD. In the method, a data processing device firstly obtains input data of an AI model; carrying out vectorization processing on the input data to obtain a first feature vector; and finally, performing data processing based on the first feature vector to obtain a processing result of the input data, wherein the data processing comprises hash processing and forward processing. According to the technical scheme of the invention, the Hash processing adopts zero-conflict Hash or limited Hash conflicts, so that the AI processor with relatively weak single instruction multi-thread (SIMT) computing power can also process or avoid the Hash conflicts, thereby improving the accuracy of AI data processing through the SIMD.

The invention discloses a high-precision and low-overhead continuous flow detection method and a high-precision and low-overhead continuous flow detection system. The method comprises the steps that firstly, a Filter module and a Detector module are arranged, and the Filter module filters intermittent flow in each time window by arranging a Bloom filter based on the time windows and identifies potential continuous flow; the Detector module is provided with a probability replacement strategy based on the completion degree of the time window, and when the Hash conflict occurs, the replacement probability is dynamically calculated and the stream which does not appear in the current time window is replaced, so that the probability that the continuous stream is wrongly replaced is reduced, and the detection precision is improved; at the end of each time window, the Filter module is updated and discontinued streams in the Detector module are cleared, while persistent streams that reach a persistence threshold are reported. According to the invention, the problems of high memory overhead and low detection precision in a high-interference traffic environment in the prior art are effectively solved, and the balance between high precision and low overhead is realized.

The invention discloses a sketch and sampling-based local differential privacy set value data aggregation method and system, and belongs to the technical field of privacy calculation and big data analysis. The method comprises the steps that a server initializes sketch structure parameters and publishes the sketch structure parameters; the user side constructs set value data, randomly selects a counter index and determines a Hash conflict set; the user side carries out randomized coding on the Hash conflict set elements according to the aggregation function type, calculates the hit rate and executes self-adaptive cutting; the user side reports the triad; and the server accumulates and updates a sketch counter, and calculates an aggregation statistical value through median estimation. According to the method, sketch sampling and lightweight reporting are used for replacing traditional multi-element disturbance, privacy budget does not need to be additionally consumed, the problem of noise accumulation is effectively solved, calculation and communication overhead is greatly reduced, and balance between privacy protection and statistical effectiveness is achieved.

This application discloses a method, apparatus, device, and storage medium for generating short links, belonging to the field of computer technology. The method includes: receiving a service request from a user terminal, the service request carrying long link information, the service request requesting the generation of N short links, where N is an integer greater than 1; generating N short codes based on a continuous factor, a random factor, and the long link information, each short code being a string, the continuous factor including a first string sequence comprising N regular first strings, and the random factor including N random second strings; performing deduplication processing on the N short codes, and generating N target short links based on the deduplication result, thereby reducing the probability of hash collisions in scenarios with a large number of short link requests and improving the effectiveness of the short links.

Embodiments of the present application provide a data storage address query method, device and equipment and medium, which are used to reduce the generation of hash conflicts in the data storage address query process. The method comprises the following steps: in response to an instruction for querying a storage address, inputting a target key value corresponding to the instruction into a plurality of linear shift registers to obtain corresponding output sequences, wherein the initial values of the linear shift registers used each time are different; determining a first hash result corresponding to the target key value according to the first M1 output sequences, and determining a second hash result corresponding to the target key value according to the last M2 output sequences; determining the position of the target hash result in a hash table according to the first hash result and the second hash result, and obtaining a corresponding target pointer from the corresponding position; and determining the storage address of the target key value from a key value table according to the target pointer.

The application discloses an Instant-NGP model compression method based on a context model, and the method comprises the following steps: constructing and training a context model customized for a hash representation of three-dimensional and two-dimensional mixing; estimating the parameter values in the hash table by using the context model; fusing the information of all index nodes generating hash conflicts in the estimated parameter values; and realizing entropy coding by using an arithmetic coding technology, so as to realize the purpose of model compression. The application introduces the context model to estimate the binary parameter values in the hash representation, so as to reduce the information entropy, and finally uses the arithmetic coding technology to realize the compression purpose.

The application discloses a low-overhead Top-k network flow high-precision extraction method, comprising: a small flow filter, which is used for filtering most small flows in a network, reducing resource overhead, and reducing a hash collision rate; and a large flow extractor, which is used for extracting Top-k flows in the network and improving the accuracy of Top-k flow identification. The method proposes an adaptive updating strategy based on a counter over-value proportion. Whenever the proportion of the number of counters exceeding a threshold value in the small flow filter is too high, each counter records whether it exceeds the threshold value in the current period through a flag bit, and then is reset to enter the next period, so that the continuous effectiveness of the small flow filter is maintained. Meanwhile, the large flow extractor is designed by using a segmented hash algorithm, multiple candidate positions are provided for incoming flows, and when all the candidate hash buckets are full, a replacement strategy is carried out through a voting mechanism, so that small flows can be kicked out as much as possible each time, and large flows can be saved in the hash buckets, and therefore the accuracy of Top-k flow extraction is improved.