52 results about "Consistent hashing" patented technology

The invention relates to the technical field of information retrieval, and discloses a distributed storage and indexing method, which comprises the following steps of: dynamically fragmenting an original file through an improved consistent Hash algorithm to generate a plurality of data blocks with timestamps; constructing a three-dimensional Bloom filter index matrix containing timestamps, data types and content features for the data blocks with the timestamps, associating a B + tree local index with an inverted global index through a hierarchical index structure, and establishing an index update priority queue by adopting a two-channel synchronization mechanism, performing real-time increment synchronization on the hotspot index through a heartbeat mechanism, performing batch synchronization on the cold data layer index according to a cold data synchronization period, predicting a data distribution probability through a distributed query statistics probability table during query, and initiating multi-path query in parallel based on probability weight, and the query path is dynamically optimized according to the node group storage medium type and the inter-node network transmission delay, so that rapid distribution adjustment and access of distributed storage are realized.

The invention relates to the technical field of industrial internet security communication, and discloses a lightweight encryption method and system for an industrial real-time data stream, and the method comprises the following steps: intercepting an original data packet, and carrying out the deep packet detection to extract a network layer quintuple and application layer metadata; matching an optimal strategy in a strategy rule set according to the feature vector and generating a scheduling instruction; in response to the instruction, calling a corresponding pre-compilation password operation pipeline to carry out differential encryption processing on the load; and constructing a fixed-length security policy head containing the algorithm template identifier, and packaging and sending the fixed-length security policy head. According to the invention, task fragmentation is carried out by using a consistent Hash technology so as to ensure processing order-preserving. By adopting a lightweight message structure without handshake negotiation and a multi-level strategy scheduling mechanism, content-based fine-grained security protection is realized, and the communication requirements of low delay and high certainty are met while the security of industrial control data is ensured.

The invention relates to an Internet of Things data security sharing method for blockchain fragmentation. The method comprises the following steps: firstly, performing identity anonymous authentication based on an auditing mechanism combining encryption packaging and zero-knowledge proof; thirdly, performing threshold slicing on the to-be-stored data by adopting an IDA slicing algorithm, performing dynamic allocation of storage nodes based on weighted consistency hash, and performing data storage and distribution based on the allocated storage nodes; afterwards, a PBFT consensus mechanism based on behavior reputation is adopted to carry out security auditing of data recovery and storage, a recovery voucher is generated, and on-chip data recovery is carried out; and finally, querying a target fragment of data distribution based on a trusted relay cross-fragment transaction mechanism, and completing cross-fragment data recovery in the target fragment. According to the invention, high efficiency and security of IoT data sharing and terminal identity privacy protection can be realized, and application requirements in a large-scale IoT environment are well met.

The invention provides a data asset management method and system based on a distributed account book technology, and relates to the technical field of data asset management, and the method comprises the steps: S1, carrying out the segmentation of a to-be-managed data asset according to a predefined fragmentation strategy, generating an encryption index corresponding to each fragment, and constructing a Merkle tree based on the data of each fragment; according to the data asset management method and system based on the distributed account book technology, a consistent hash + virtual node fragmentation strategy is adopted, so that data distribution is balanced, full-amount remapping is not needed when nodes are newly added / removed, and the system maintenance cost is greatly reduced. According to the method, fragments are divided into multiple batches based on a load balancing algorithm, PBFT consensus requests are initiated in parallel, TPS is improved to a multiple level from a single instance mode through parallel block merging, and dynamic balance of high throughput and low acknowledgement delay is achieved by means of a lightweight BFT and parallel PBFT switching mechanism triggered by a threshold value. And it is ensured that the system always maintains the optimal performance during a service peak period and resource fluctuation.

The application belongs to the field of data storage, and discloses a directory distribution method, an electronic device and a computer readable storage medium. The method comprises the following steps: inserting a to-be-created directory item into a virtual directory of a virtual set of a distributed file system based on a skip consistent hashing algorithm; if the storage amount of the virtual directory is greater than a preset storage threshold, determining whether the number of sets of the virtual set in the distributed file system is less than a preset set threshold; if yes, splitting a new directory set from the virtual set and allocating a metadata node to the new directory set; if no, determining whether the number of directories of the virtual directory is less than a preset directory threshold; if the number of directories is less than the preset directory threshold, performing directory splitting based on a global bitmap of the virtual set where the virtual directory is located; and if the number of directories is greater than or equal to the preset directory threshold, controlling the directory expansion and the directory splitting to be alternately split. The application aims to solve the technical problem that the creation of a directory in a distributed file system is not efficient enough.

The present application relates to a device scheduling access method and system, a readable storage medium, and a program product. The method comprises: in response to a session request sent by a communication device, performing consistent hashing processing on a device code of the communication device by means of a target forwarding server to obtain a hash value; on the basis of a correspondence between the hash value and a signaling service identifier, determining a target signaling service identifier corresponding to the hash value; and distributing the session request to a target signaling server corresponding to the target signaling service identifier. By using the method, on the basis a finer-grained load policy, requests of a same communication device can be ensured to be all processed on a same signaling service instance, thereby implementing platform statelessness and efficient load balancing.

The application discloses a message distribution method and device, equipment and product, and relates to the technical field of communication. The message distribution method is applied to a non-central architecture publishing and subscribing system, the publishing and subscribing system comprises a subscriber cluster and an arbitration center, and the message distribution method comprises the following steps: determining the subscription priority of a plurality of subscribers in the subscriber cluster corresponding to a subscription topic based on a consistent hashing algorithm and the subscription topic; determining a first type of subscriber according to a preset lease mechanism based on the arbitration center and the subscription priority; and distributing the message of the subscription topic to the first type of subscriber, so that the first type of subscriber performs consumption. Through the above scheme, the efficiency of message distribution can be effectively improved, and efficient load balancing without centralization is realized.

The invention discloses a DNS (Domain Name Server) cache optimization system based on multi-node collaboration, which is suitable for public DNS, operator recursion, CDN (Content Delivery Network) scheduling and cloud native scenes. The system is composed of an edge recursion node, a shared second-level cache layer, a cooperative forwarding and near-source rollback module, a distributed active refreshing and lease cooperation module, a cache key generation and routing module and a toughness mechanism module. The domain name and the client network segment are taken as main keys, and the ECS and the two-stage consistent Hash are combined to realize request viscosity and fragmentation stability; when a local miss occurs, preferentially forwarding to a same network segment / same city-same operator / same region collaborative node, and then returning to share cache and authority; a coordination node is elected and refreshed according to domain name fragments, active refreshing of hotspots and immediate entries is driven by lease + priority, and group frightening is avoided through push / pull diffusion; toughness capabilities such as request merging, fusing degradation, health detection and the like are provided, and automatic near-source fallback and removal of abnormities are achieved. According to the method, the hit rate and the first packet delay can be remarkably improved.

The invention discloses an industrial digital twinborn model updating method based on consistent Hash cooperative caching, which comprises the following steps: performing digital twinborn synchronization on multivariate perception data of equipment of different production lines, constructing a group learning digital twinborn model, clustering similar industrial equipment through semantic feature vectors of the industrial equipment in the digital twinborn model of each production line, and establishing a group learning digital twinborn model; and constructing a Hash ring corresponding to each production line based on a consistent Hash algorithm, and merging or splitting the cluster virtual nodes according to the current semantic feature vectors and loads of the cluster virtual nodes when the digital twin model is updated each time, thereby realizing the updating of the digital twin model. According to the method, the digital twin model is updated through combination or splitting of cluster virtual nodes on a hash ring, so that redundant resources are compressed, and resource scheduling is realized.

The embodiment of the specification discloses a communication method and device of a distributed device registration center, an apparatus and a medium, the distributed device registration center comprising four-layer load balancing software, a network component, a device shadow component and seven-layer load balancing software. The network component can be used for connection between a platform and a device, and maintained state information is persisted to a database. The information is updated when the device is disconnected or reconnected. Since the four-layer load balancing software uses a polling strategy to distribute requests, each network component is equally replaceable, and there is no single-point risk. Meanwhile, each device shadow component specifically manages the shadow of which device, which can depend on the ID of the device and the number of device shadow components in the cluster. According to the consistent hashing algorithm, when the device is determined and the number of device shadow components in the cluster is determined, a uniquely determined device shadow component can be obtained. When the number of device shadow components in the cluster changes, the maintained relationship can be recalculated, and there is no single-point risk.

The invention discloses an industrial monitoring intelligent alarm method, device and system based on dynamic speech synthesis, and a storage medium. The method comprises the following steps: receiving structured alarm event data of a front end; generating a to-be-synthesized voice text according to the data and a preset template; calling a locally deployed voice synthesis engine to process the voice text to generate voice audio data; and transmitting the audio data stream to a front end for playing. Wherein when the speech synthesis engine executes the synthesis task, the index file is compiled in real time to support hot update of the term library. And meanwhile, in the speech synthesis process, a consistent Hash algorithm is adopted to allocate tasks and automatically expand a process pool when the load is high, and meanwhile, a three-stage fault-tolerant mechanism of synthesis overtime switching, template error degradation and engine crash restarting is provided. The problems that an existing industrial voice alarm scheme is poor in flexibility and intranet safety, term hot update, high concurrency and low delay cannot be considered at the same time are solved, and real-time and dynamic generation and high-reliability broadcasting of thousand-level concurrency alarm voice in the domestic industrial intranet environment are achieved.

The embodiment of the invention provides a data processing method, electronic equipment, a storage medium and a program product. The data processing method comprises the following steps: acquiring a data volume relationship of to-be-Hash distributed data corresponding to all associated nodes; and balancing the data volume of the to-be-Hash distributed data corresponding to the associated nodes based on the data volume relationship, so that the data volume of the to-be-Hash distributed data of each associated node is kept within a preset range threshold. According to the scheme of the embodiment, it is guaranteed that the data size corresponding to each association node is kept uniform, so that uniform hash distribution is achieved on the basis of consistent hash distribution, and the situation that some association nodes in one association subsystem are overloaded and some association nodes are overloaded is avoided.

The application discloses an information security data storage system and relates to the technical field of data storage.The application solves the technical problem of avoiding the problem of resource waste or performance bottleneck caused by one-size-fits-all storage by intelligently selecting local storage or distributed storage mode according to the type, capacity size and delay requirement of data.The application dynamically selects local or distributed storage based on data capacity and delay requirement, stores small-capacity and low-delay data locally to reduce network overhead, stores large-capacity and high-delay data in a distributed manner to improve scalability, improves resource utilization, automatically deletes temporary data, combines high-frequency / low-frequency data differentiation processing, takes into account data security and space optimization, dynamically adapts different capacity data through a dynamic sharding strategy, ensures balanced distribution of data through consistent hashing and virtual nodes, and realizes dynamic load adjustment through cold / hot data layering and node performance grading, thereby improving access speed and system stability.

The application provides a state compression consistent hashing method supporting arbitrary removal of an evidence storage node, comprising: constructing a state compression layer for recording a removal state of the storage node; when a storage node is removed, updating the removal state through the state compression layer with constant time complexity; based on the removal state recorded by the state compression layer, addressing a data key, and based on a stateless consistent hashing engine, performing primary mapping on the data key to obtain a first node number; when the storage node corresponding to the first node number is in a working state, outputting the first node number as an addressing result; or when the storage node corresponding to the first node number is in a removal state, triggering secondary mapping, recalculating the first node number in a current working storage node set, and outputting a second node number as an addressing result. The method realizes high efficient search performance and low memory overhead while supporting arbitrary random removal of the storage node.

The invention discloses a distributed complex task parallel processing system and method. The system comprises a global task scheduling and coordination module, a task division and distribution module, distributed parallel computing nodes, a result aggregation module and a strong consistency verification and fault tolerance module. According to the system, dynamic fragmentation and node mapping of tasks are achieved by calculating task feature fingerprints and adopting a consistent Hash algorithm, and only data migration in the minimum range is needed during node capacity expansion and contraction; by setting a blocking fence synchronization point and a periodic verification mechanism, high consistency and reproducibility of a global task result are realized. And when node faults or inconsistent results are detected, the system automatically rolls back to the last successful fence check point and performs task rescheduling, so that the completeness and stability of calculation are ensured. The method has remarkable advantages in concurrency performance, consistency guarantee and fault-tolerant recovery, and is suitable for scenes such as large-scale data analysis, scientific simulation and real-time decision calculation.

The application provides a stock selection system based on a distributed efficient index formula, comprising a front-end agent service module, a plurality of sharding calculation service modules and a data source service module. The front-end agent service module receives a user request containing a plate factor and an index formula, acquires a benchmark stock pool list requiring stock selection according to the plate factor, divides the benchmark stock pool list requiring stock selection into a plurality of shards by taking the modulus of an incremental internal code and the total number of shards, and generates a corresponding shard calculation request; a shard calculation node management module in the front-end agent service module distributes the request to the corresponding shard calculation service module according to the shard identifier by using an adaptive consistent hashing algorithm. The application separates the front-end agent and the shard calculation by combining the MapReduce task distribution and the result aggregation mechanism, improves the real-time performance and the concurrent processing capability of stock selection calculation, dynamically adjusts the distribution of virtual nodes and entity nodes by using the adaptive consistent hashing strategy, solves the problem of unbalanced sharding, and supports node elastic scaling.

This invention discloses a task allocation method and device. The method includes: if the target node is the master node, then when the periodic inspection time arrives, acquiring information on defined tasks, information on delegated tasks, information on a first set of nodes with delegated tasks, and information on a second set of available nodes; if the information on defined tasks differs from the information on delegated tasks, and / or the information on the first set differs from the information on the second set, then acquiring a first latest set of available nodes and a first latest set of full tasks; based on the first latest set of available nodes and the first latest set of full tasks, performing a task reallocation operation using a consistent hashing algorithm to obtain a first mapping relationship between nodes and delegated tasks; and writing the first mapping relationship into a database node. This task allocation method reduces task execution latency, improves the throughput of the distributed scheduling system, and enhances the reliability and resource utilization during the task allocation process.

The application discloses an audio integrated network resource virtualization method, and belongs to the technical field of audio network resource management, and comprises the following steps: constructing a distributed virtual switching layer of a covering subnet, embedding a virtual switch instance into a kernel state network stack of each physical host in a cluster, and dynamically mapping cloud hosts in the subnet to the physical host group through a consistent hashing algorithm; when the cloud host sends an audio data packet, the source physical host virtual switch instance performs the following steps: extracting a DSCP field and marking it as EF, encapsulating through a GRE-over-UDP tunnel, and replacing a target MAC address with a virtual MAC address of a next-hop virtual switch instance; after the virtual switch instance of the target physical host is unpacked, the audio packet is delivered to the target cloud host based on a virtual port forwarding table. Through the distributed virtual switching layer and the consistent hashing mapping mechanism, the method realizes efficient virtualization transmission of audio data, supports QoS guarantee and low-delay transmission, and is suitable for audio network resource management in a large-scale cloud environment.

This invention discloses a high-concurrency access control system and method for multi-terminal collaboration in construction management. The system includes: a terminal access authentication module, which performs identity authentication and role classification on multi-terminal devices, derives corresponding terminal classifications, and assigns corresponding role weight coefficients; a dynamic hierarchical rate limiting module, which calculates the token generation rate and token bucket capacity based on a preset dynamic hierarchical token bucket algorithm and performs token verification and rate limiting control on concurrent access requests from multiple terminals according to the role weight coefficients; a distributed scheduling module, which uses consistent hashing distributed scheduling rules to distribute access requests that pass rate limiting verification, and simultaneously completes multi-terminal session state synchronization based on a distributed session cluster; and a data conflict resolution module, which performs two-factor conflict detection on triggered operations on the same data item, performs conflict resolution, and completes data read / write and returns an access response. Its beneficial effects are: making differentiated rate limiting more reasonable and automatically adapting to on-site network fluctuations.

Techniques are described for enabling uninterrupted services by computing resources on nodes of a consistency hash ring (CHR) while adding or removing nodes (i.e., making changes) of the CHR. In some embodiments, a duplicate of the existing CHR (i.e., old version) is created to become a new version for performing the changes. Two versions of consistent hash rings (CHRs) co-exist during the transition period of making changes. In some embodiments, computing resources on the nodes of these CHRs perform version upgrades and data migration while continuing to service client requests without interruption.

The invention relates to the technical field of distributed storage, and particularly discloses a method for distributing distributed storage data positions, which comprises the following steps of: dividing a disk of a node into partitions with fixed sizes, and organizing the partitions into storage buckets; the client provides a data block to be written, calculates a hash value of the data block, and distributes the data block to a storage bucket through a consistent hash algorithm; nodes corresponding to the partitions in the storage bucket form a negotiation group, the client broadcasts a write-in request to the negotiation group, the negotiation group executes a rapid negotiation step according to the node state and generates a negotiation proposal, the negotiation group sends the negotiation proposal to the client, and the client generates a negotiation result according to the negotiation proposal, the response speed and the write-in request; and the client broadcasts the negotiation result to the negotiation group and sends a write-in instruction to the node according to the negotiation result, and the node executes write-in operation. The data storage positions are dynamically allocated according to the resource conditions of the nodes, so that the resource utilization rate and the data writing performance of the distributed storage system are improved.

A method and device for scaling a distributed database and a storage medium, comprising: in the case of meeting a scaling condition, scaling the distributed database using a consistent hashing algorithm; during the scaling, suspending a write request from a client for a target physical node; obtaining feature information of the suspended write request according to a preset strategy; in the case of the obtained feature information reaching a feature threshold, suspending the scaling, and processing a new write request from the client for the target physical node and the suspended write request whose feature information reaches the feature threshold, until the scaling is resumed in the case of re-meeting a scaling preparation condition. The embodiments of the present disclosure enable the distributed database to still provide normal services during scaling, thereby avoiding service interruption of the distributed database and improving service availability of the business distributed database.

Systems, methods, and network attached storage nodes for secure peer-to-peer file sharing using consistent hashing for distributed ownership data. A plurality of network attached storage nodes may be interconnected by a network and configured for peer-to-peer communication without a centralized server. When a node receives a user data file, it divides the file into data chunks and determines hashes for those data chunks. It uses the chunk hashes to map the data chunks to other nodes for storage. It also sends ownership data for the file that includes an owner identifier, a hash key, the set of chunk hashes.

This application discloses a distributed device time-series data storage method, device, and medium, relating to the field of data storage technology. The method includes: receiving time-series data generated by an IoT device in real time, and acquiring the data generation rate of the IoT device and the real-time storage status of the distributed storage nodes; dynamically planning the write path to the target storage node corresponding to the time-series data based on the data generation rate and real-time storage status; performing data sharding on the time-series data based on a preset consistent hashing algorithm, and writing the time-series data to the target storage node through the target storage node write path based on the data sharding results; creating a backup copy of the time-series data through a multi-replica mechanism, and monitoring the health status of the target storage node to switch the access path of the time-series data to the backup copy when the target storage node fails.

The invention discloses a multi-modal high-speed data storage method for a potassium collecting machine, which relates to the technical field of data storage, and comprises the following steps: selecting key value storage according to the storage demand of multi-modal data of the potassium collecting machine, and defining the structure of a key matched with the type of the multi-modal data, each multi-modal data item needing to be stored has a unique corresponding key; uniformly distributing the multi-modal data items on different nodes in a computer cluster matched with the potassium collecting machine by using a consistent Hash algorithm so as to realize load balancing; after the multi-modal data item is allocated to the target node, writing an operation log record of the multi-modal data item into a disk; writing the multi-modal data item into a memory cache; and finally, flashing the multi-modal data in the memory cache into a disk according to a preset period. The number of the virtual nodes can be intelligently adjusted according to the change of the system, so that better load balancing and system performance are realized, and the flexibility, the stability and the reliability of the system are improved.

The invention discloses a distributed real-time database high availability method, system and device of a new energy centralized control station system and a storage medium. Based on a distributed architecture, a real-time data writing strategy, a real-time data reading strategy of the distributed architecture, a high-availability strategy after host exit of the distributed architecture and a high-availability strategy after host recovery of the distributed architecture are constructed, and respective storage nodes and hot standby nodes are created for a plurality of hosts to form a complete real-time library. According to the method, the acquisition, storage and reading of the large-capacity real-time database of the new energy centralized control station system are realized by utilizing load balancing, consistent hash, node heartbeat monitoring and failover, the high availability of real-time data is realized, and the reliability of the whole monitoring system is improved.

A business rule hot update method based on quantitative analysis and dynamic arbitration, relating to the field of distributed computing, is disclosed. The method includes: quantifying the impact of data changes on business metrics through rule impact strength network analysis, triggering incremental recalculation only for high-impact changes; routing recalculation tasks to a fixed subset of arbitration nodes using consistent hashing, ensuring strong consistency of results through multi-node parallel computing and load-aware arbitration; storing both arbitration and old versions of the results, and dynamically adjusting the canary release ratio based on a PID feedback controller according to real-time risk indicators to achieve adaptive risk control and second-level automatic rollback. This invention solves the problems of service interruption, data inconsistency, and high release risk in traditional solutions, achieving minute-level, accurate, and secure hot updates of business rules, making it particularly suitable for high-requirement scenarios such as e-commerce and finance.

The invention provides an alarm computing load balancing method, which belongs to the technical field of cloud computing and distributed monitoring operation and maintenance, adopts a distributed architecture, and comprises a plurality of Prometheus examples, a scheduling management module, a task distributor, an equal division algorithm executor and a shared storage component. The scheduling management module dynamically fragments the alarm rules according to rule types, data source features or time dimensions, and adjusts the number and size of fragments according to real-time loads. And the equal division algorithm executor combines the load, the performance index and the resource utilization rate of each instance, adopts strategies such as consistent Hash, polling or weighted polling to calculate an assignment relationship, and supports newly-added rule allocation and fragment migration. Dynamic management and real-time synchronization of rules are achieved through the shared storage component, stateless deployment of Prometheus instances is achieved, requests are distributed through the load balancer, and therefore the load balancing capacity, the rule management efficiency and the alarm timeliness are improved.

The invention discloses a flow load balancing algorithm for a multi-node encryption device driven by a control machine, which aims at encryption service requirements in key fields, adopts a'centralized decision-making and distributed execution 'framework and comprises the control machine, a flow distributor and an encryption node cluster. The control machine periodically collects node multi-dimensional state data and calculates a comprehensive load evaluation value; extracting encrypted request features, calculating the matching degree of nodes and requests, and dynamically allocating traffic (state keeping requests are forwarded by using consistent Hash, and stateless requests are allocated according to probability). The system has fault detection isolation and gray recovery mechanisms, and algorithm parameters are optimized through a feedback learning module. According to the algorithm, the resource utilization rate and throughput are improved, the response time delay is reduced, the system reliability and adaptability are enhanced, and the high-reliability and high-throughput encryption service requirements are met.

The application discloses a kind of based on erasure code's storage optimization method for decentralized storage system, comprising: file stream is divided into blocks and is organized as basic Merkle directed acyclic graph, and the file stream of block is encoded with erasure code, and EC-Merkle DAG is constructed;The leaf block of EC-Merkle DAG and basic Merkle directed acyclic graph is stored by consistent hashing, and the non-leaf block of both is stored to all nodes by replication;Metadata consensus is carried out by the metadata consensus mechanism based on Merkle-CRDT;The identifier set of leaf data block is obtained based on basic Merkle directed acyclic graph, and further file acquisition operation is completed, if there is the storage node of leaf data block downtime or data damage, then file repair is carried out.The application can improve the storage efficiency of data in decentralized storage scene, and ensure the integrity and security of data.