88 results about "Consistent hashing" patented technology

The invention relates to the technical field of information retrieval, and discloses a distributed storage and indexing method, which comprises the following steps of: dynamically fragmenting an original file through an improved consistent Hash algorithm to generate a plurality of data blocks with timestamps; constructing a three-dimensional Bloom filter index matrix containing timestamps, data types and content features for the data blocks with the timestamps, associating a B + tree local index with an inverted global index through a hierarchical index structure, and establishing an index update priority queue by adopting a two-channel synchronization mechanism, performing real-time increment synchronization on the hotspot index through a heartbeat mechanism, performing batch synchronization on the cold data layer index according to a cold data synchronization period, predicting a data distribution probability through a distributed query statistics probability table during query, and initiating multi-path query in parallel based on probability weight, and the query path is dynamically optimized according to the node group storage medium type and the inter-node network transmission delay, so that rapid distribution adjustment and access of distributed storage are realized.

The invention provides a multi-scene access control dynamic verification method and system based on a permission tree, and relates to the technical field of scene access, and the method comprises the steps: receiving a user resource access request, obtaining a user permission tree, obtaining a mapping rule according to a scene identifier, screening to generate a primary screening permission node set, carrying out the grouping processing through employing a consistent Hash algorithm, and obtaining a primary screening permission node set; and generating a scenarized permission tree through reachability analysis, merging and reconstruction, performing hierarchical verification on identity validity, scene legality and resource access permission on the request, and generating an access control strategy. The method can dynamically adapt to authority requirements of different scenes, and the accuracy and flexibility of resource access control are improved.

Systems, methods, and network attached storage nodes for peer-to-peer file sharing using consistent hashing for distributing data among storage nodes. A plurality of network attached storage nodes may be interconnected by a network and configured for peer-to-peer communication without a centralized server. When a node receives a user data file, it divides the file into data chunks and determines hashes for those data chunks. It uses consistent hashing of the chunk hashes and storage node addresses to map the data chunks to other nodes for distributed storage.

The present application relates to the technical field of middleware, and in particular to a cache data semi-synchronization method, comprising comprises the following steps: writing and running a distributed cache proxy layer service using a preset programming language, receiving a read / write command request from an application client, parsing a Key consistent hashing value and sending same to a rear-end Redis sharding node. A master instance parses a message packet, marks a write operation and records same in an AOF file. Upon write to disk, the master instance sends an AOF file transfer command to a slave instance node, and the slave instance acknowledges reception. When receiving a reception response, the master instance updates an acknowledgment offset of the slave node and responds to the client. When network delay is high, data of a master node and a slave node is marked as "asynchronous", and it is automatically downgraded to asynchronous replication. According to the present application, by means of a self-developed Redis kernel, an asynchronous ack mechanism is used to optimize redis to achieve master-slave semi-synchronization, thereby enhancing data reliability of a single point of failure, and eliminating the influence introduced by a native Redis asynchronous replication mechanism such as data delay and data consistency.

The invention discloses a computer self-check cycle optimization method based on the Internet of Things in the technical field of server maintenance, and relates to the technical field of server maintenance. Server node operation state data is collected in real time, a dynamic load evaluation model is constructed, and a comprehensive load index is calculated; and dynamically dividing performance states based on the threshold values L1 and L2, and adjusting a self-checking period and a module priority. Calculating a single-node risk index Ri by adopting a fault risk model, generating a fault generalization risk factor R1 through a probability graph model in combination with network topology, and triggering a linkage mechanism; dividing and dynamically adjusting a self-checking period according to the early warning level; during advanced early warning, redundant resources are distributed to preferentially process key faults, edge calculation is combined to carry out data cleaning and standardization, distributed management is realized through a consistent Hash algorithm, and finally, a visual interface is used for displaying an optimized scheduling result, so that a closed-loop optimization system of data acquisition, risk prediction and resource scheduling is formed.

The invention discloses a synchronization method and device for storage metadata, electronic equipment and a storage medium, and relates to the technical field of computers, a consistent Hash algorithm is adopted to realize mapping of data fragments and storage nodes, a monitoring node maintains the relation, a data change event is directionally sent to a corresponding partition, and the data change event is stored in the corresponding partition. Meanwhile, the Bloom filter is used for pre-checking events to filter invalid information, only local related changes are processed, the check value is transmitted, and the memory area is pre-registered, so that the data interaction mode between the nodes is optimized, and the defects of a full-connection mode are avoided; the technical problems that in an existing block storage metadata synchronization mechanism, due to the fact that a full connection mode is not optimized for node scale extension, the network connection number is increased in an exponential level mode, the system overhead is increased, and the cluster expansion capacity is limited can be solved, and the purposes of reducing the system overhead, improving the cluster expansion capacity and improving the cluster expansion efficiency are achieved. And the performance of the distributed block storage system is optimized.

The invention relates to the technical field of data processing, and provides a task scheduling method and system applied to agents, and the method comprises the steps: obtaining a plurality of to-be-allocated tasks of a multi-agent platform, a plurality of historical tasks processed by each agent, and a current task being processed; quantifying the initial priority of each to-be-allocated task; obtaining an initial adaptation degree of each agent to each to-be-allocated task; performing Hash ring mapping through a consistent Hash algorithm according to the task type to obtain a mapping position of each task, and obtaining a corresponding node for each agent; performing Hash ring mapping on each to-be-allocated task to obtain a processing matching degree of each agent to each to-be-allocated task; and constructing a scheduling optimization function for all the current to-be-allocated tasks and obtaining an optimal solution to realize task scheduling of the intelligent agent. The invention aims to solve the problem that the task processing efficiency is reduced because the agent task scheduling does not consider the previous task processing of each agent.

The invention discloses a high-dimensional data retrieval method based on distributed computing and mixed indexing. The method comprises the following steps: step 1, data preprocessing; step 2, cluster generation; step 3, index construction; step 4, consistent Hash: distributing each cluster to a virtual node of a Hash ring, dynamically adjusting data distribution on the virtual node according to a load strategy, and mapping the virtual node to a physical server; and step 5, hierarchical aggregation: mapping a query request to a node and adjacent nodes on a hash ring through a hash function, performing candidate set calculation in parallel, merging candidate sets sent by all the nodes by a Spark center node, and finally optimizing a result. According to the method, high-recall-rate and low-delay distributed retrieval is achieved through mixed index construction, dynamic consistency hash allocation and hierarchical aggregation strategies, meanwhile, system load balancing and index updating efficiency are guaranteed, and the method is suitable for large-scale high-dimensional data scenes.

The invention relates to the technical field of industrial internet security communication, and discloses a lightweight encryption method and system for an industrial real-time data stream, and the method comprises the following steps: intercepting an original data packet, and carrying out the deep packet detection to extract a network layer quintuple and application layer metadata; matching an optimal strategy in a strategy rule set according to the feature vector and generating a scheduling instruction; in response to the instruction, calling a corresponding pre-compilation password operation pipeline to carry out differential encryption processing on the load; and constructing a fixed-length security policy head containing the algorithm template identifier, and packaging and sending the fixed-length security policy head. According to the invention, task fragmentation is carried out by using a consistent Hash technology so as to ensure processing order-preserving. By adopting a lightweight message structure without handshake negotiation and a multi-level strategy scheduling mechanism, content-based fine-grained security protection is realized, and the communication requirements of low delay and high certainty are met while the security of industrial control data is ensured.

The invention relates to an Internet of Things data security sharing method for blockchain fragmentation. The method comprises the following steps: firstly, performing identity anonymous authentication based on an auditing mechanism combining encryption packaging and zero-knowledge proof; thirdly, performing threshold slicing on the to-be-stored data by adopting an IDA slicing algorithm, performing dynamic allocation of storage nodes based on weighted consistency hash, and performing data storage and distribution based on the allocated storage nodes; afterwards, a PBFT consensus mechanism based on behavior reputation is adopted to carry out security auditing of data recovery and storage, a recovery voucher is generated, and on-chip data recovery is carried out; and finally, querying a target fragment of data distribution based on a trusted relay cross-fragment transaction mechanism, and completing cross-fragment data recovery in the target fragment. According to the invention, high efficiency and security of IoT data sharing and terminal identity privacy protection can be realized, and application requirements in a large-scale IoT environment are well met.

This application relates to the field of cloud storage technology. Specifically, it discloses a video surveillance storage system and method based on distributed cloud storage. It collects surveillance video streams through cameras, performs time segmentation on the surveillance video streams based on the video retrieval query status of users to form multiple video data blocks, and then constructs a video distributed cloud storage architecture. It uses the consistent hashing algorithm to determine the storage locations of each video data block in the distributed cloud storage, and monitors the usage frequency of each video data block in real time to dynamically adjust the storage priorities of each video data block. This application can not only effectively avoid the risk of single point of failure, improve the reliability and fault tolerance of storage, but also dynamically adjust the storage priorities of video data according to their usage frequencies, realizing efficient management and access of data.

The invention discloses an optical transmission system management platform based on distributed storage and multi-protocol adaptation, and relates to the technical field of optical transmission network operation and maintenance. The platform comprises a data acquisition module, a distributed storage module, a data processing module, an intelligent analysis module and a user interaction module. And the data acquisition module is compatible with SNMP, NETCONF, TL1 and private protocols through a dynamic protocol analysis engine to realize unified acquisition of heterogeneous device data. The distributed storage module adopts a consistent Hash algorithm to perform fragmentation storage, and combines a multi-copy mechanism and a log structure to merge tree index optimization, thereby improving storage reliability, load balance and query efficiency.

The present invention discloses a distributed intelligent recognition and processing system for collaborative aerial images of a cluster of unmanned aerial vehicles (UAVs), which belongs to the technical field of UAV image processing. The system specifically comprises: UAVs collect aerial images according to a geographical division scheme and store them in a local edge cache, and then split the aerial image data into two-layer tasks according to color and texture features and upload them to a regional cache; each node obtains data from a three-level cache system, completes target recognition using an intelligent collaborative recognition algorithm after preliminary processing, and stores the results in a global cache; the central node monitors the load weight of each node in real time, and allocates new tasks using a weighted minimum connection number algorithm; the central node aggregates the recognition results in the global cache according to the grid ID, eliminates splicing dislocations through a consistent hashing algorithm, and generates an aerial image recognition report output; the present invention effectively solves the problems of low processing efficiency and large collaborative errors in a centralized architecture, and improves the efficiency and accuracy of aerial image processing of UAV clusters.

The invention provides a data asset management method and system based on a distributed account book technology, and relates to the technical field of data asset management, and the method comprises the steps: S1, carrying out the segmentation of a to-be-managed data asset according to a predefined fragmentation strategy, generating an encryption index corresponding to each fragment, and constructing a Merkle tree based on the data of each fragment; according to the data asset management method and system based on the distributed account book technology, a consistent hash + virtual node fragmentation strategy is adopted, so that data distribution is balanced, full-amount remapping is not needed when nodes are newly added / removed, and the system maintenance cost is greatly reduced. According to the method, fragments are divided into multiple batches based on a load balancing algorithm, PBFT consensus requests are initiated in parallel, TPS is improved to a multiple level from a single instance mode through parallel block merging, and dynamic balance of high throughput and low acknowledgement delay is achieved by means of a lightweight BFT and parallel PBFT switching mechanism triggered by a threshold value. And it is ensured that the system always maintains the optimal performance during a service peak period and resource fluctuation.

The application belongs to the field of data storage, and discloses a directory distribution method, an electronic device and a computer readable storage medium. The method comprises the following steps: inserting a to-be-created directory item into a virtual directory of a virtual set of a distributed file system based on a skip consistent hashing algorithm; if the storage amount of the virtual directory is greater than a preset storage threshold, determining whether the number of sets of the virtual set in the distributed file system is less than a preset set threshold; if yes, splitting a new directory set from the virtual set and allocating a metadata node to the new directory set; if no, determining whether the number of directories of the virtual directory is less than a preset directory threshold; if the number of directories is less than the preset directory threshold, performing directory splitting based on a global bitmap of the virtual set where the virtual directory is located; and if the number of directories is greater than or equal to the preset directory threshold, controlling the directory expansion and the directory splitting to be alternately split. The application aims to solve the technical problem that the creation of a directory in a distributed file system is not efficient enough.

The present application relates to a device scheduling access method and system, a readable storage medium, and a program product. The method comprises: in response to a session request sent by a communication device, performing consistent hashing processing on a device code of the communication device by means of a target forwarding server to obtain a hash value; on the basis of a correspondence between the hash value and a signaling service identifier, determining a target signaling service identifier corresponding to the hash value; and distributing the session request to a target signaling server corresponding to the target signaling service identifier. By using the method, on the basis a finer-grained load policy, requests of a same communication device can be ensured to be all processed on a same signaling service instance, thereby implementing platform statelessness and efficient load balancing.

The application discloses a message distribution method and device, equipment and product, and relates to the technical field of communication. The message distribution method is applied to a non-central architecture publishing and subscribing system, the publishing and subscribing system comprises a subscriber cluster and an arbitration center, and the message distribution method comprises the following steps: determining the subscription priority of a plurality of subscribers in the subscriber cluster corresponding to a subscription topic based on a consistent hashing algorithm and the subscription topic; determining a first type of subscriber according to a preset lease mechanism based on the arbitration center and the subscription priority; and distributing the message of the subscription topic to the first type of subscriber, so that the first type of subscriber performs consumption. Through the above scheme, the efficiency of message distribution can be effectively improved, and efficient load balancing without centralization is realized.

The invention discloses a DNS (Domain Name Server) cache optimization system based on multi-node collaboration, which is suitable for public DNS, operator recursion, CDN (Content Delivery Network) scheduling and cloud native scenes. The system is composed of an edge recursion node, a shared second-level cache layer, a cooperative forwarding and near-source rollback module, a distributed active refreshing and lease cooperation module, a cache key generation and routing module and a toughness mechanism module. The domain name and the client network segment are taken as main keys, and the ECS and the two-stage consistent Hash are combined to realize request viscosity and fragmentation stability; when a local miss occurs, preferentially forwarding to a same network segment / same city-same operator / same region collaborative node, and then returning to share cache and authority; a coordination node is elected and refreshed according to domain name fragments, active refreshing of hotspots and immediate entries is driven by lease + priority, and group frightening is avoided through push / pull diffusion; toughness capabilities such as request merging, fusing degradation, health detection and the like are provided, and automatic near-source fallback and removal of abnormities are achieved. According to the method, the hit rate and the first packet delay can be remarkably improved.

The invention discloses an industrial digital twinborn model updating method based on consistent Hash cooperative caching, which comprises the following steps: performing digital twinborn synchronization on multivariate perception data of equipment of different production lines, constructing a group learning digital twinborn model, clustering similar industrial equipment through semantic feature vectors of the industrial equipment in the digital twinborn model of each production line, and establishing a group learning digital twinborn model; and constructing a Hash ring corresponding to each production line based on a consistent Hash algorithm, and merging or splitting the cluster virtual nodes according to the current semantic feature vectors and loads of the cluster virtual nodes when the digital twin model is updated each time, thereby realizing the updating of the digital twin model. According to the method, the digital twin model is updated through combination or splitting of cluster virtual nodes on a hash ring, so that redundant resources are compressed, and resource scheduling is realized.

The invention discloses a TCP data processing system and method across public and private networks. The system comprises a public network terminal, a plurality of servers of a private network and a management platform crossing the public network and the private network, the platform comprises a first agent module, a distribution management module and a plurality of second agent modules. The first agent module obtains TCP-based first data of the terminal, packages the first data into HTTP second data and transmits the HTTP second data to the distribution management module; the distribution management module selects a target server according to a load balancing strategy, binds the terminal and the server through a consistent Hash algorithm, and transmits second data to a corresponding second agent module; the target second agent module analyzes the first data, transmits the first data to the server, packages TCP third data returned by the server into HTTP fourth data and transmits the HTTP fourth data back; and the first agent module analyzes third data and feeds back the third data to the terminal. According to the method and the device, the technical problem that cloud card reading and other services depending on TCP direct connection of the public network cannot run normally in a scene that an enterprise closes part of public network ports for network security is solved.

The embodiment of the specification discloses a communication method and device of a distributed device registration center, an apparatus and a medium, the distributed device registration center comprising four-layer load balancing software, a network component, a device shadow component and seven-layer load balancing software. The network component can be used for connection between a platform and a device, and maintained state information is persisted to a database. The information is updated when the device is disconnected or reconnected. Since the four-layer load balancing software uses a polling strategy to distribute requests, each network component is equally replaceable, and there is no single-point risk. Meanwhile, each device shadow component specifically manages the shadow of which device, which can depend on the ID of the device and the number of device shadow components in the cluster. According to the consistent hashing algorithm, when the device is determined and the number of device shadow components in the cluster is determined, a uniquely determined device shadow component can be obtained. When the number of device shadow components in the cluster changes, the maintained relationship can be recalculated, and there is no single-point risk.

Systems, methods, and network attached storage nodes for secure peer-to-peer file sharing using consistent hashing for distributed ownership data. A plurality of network attached storage nodes may be interconnected by a network and configured for peer-to-peer communication without a centralized server. When a node receives a user data file, it divides the file into data chunks and determines hashes for those data chunks. It uses the chunk hashes to map the data chunks to other nodes for storage. It also sends ownership data for the file that includes an owner identifier, a hash key, the set of chunk hashes.

The invention discloses an industrial monitoring intelligent alarm method, device and system based on dynamic speech synthesis, and a storage medium. The method comprises the following steps: receiving structured alarm event data of a front end; generating a to-be-synthesized voice text according to the data and a preset template; calling a locally deployed voice synthesis engine to process the voice text to generate voice audio data; and transmitting the audio data stream to a front end for playing. Wherein when the speech synthesis engine executes the synthesis task, the index file is compiled in real time to support hot update of the term library. And meanwhile, in the speech synthesis process, a consistent Hash algorithm is adopted to allocate tasks and automatically expand a process pool when the load is high, and meanwhile, a three-stage fault-tolerant mechanism of synthesis overtime switching, template error degradation and engine crash restarting is provided. The problems that an existing industrial voice alarm scheme is poor in flexibility and intranet safety, term hot update, high concurrency and low delay cannot be considered at the same time are solved, and real-time and dynamic generation and high-reliability broadcasting of thousand-level concurrency alarm voice in the domestic industrial intranet environment are achieved.

The embodiment of the invention provides a data processing method, electronic equipment, a storage medium and a program product. The data processing method comprises the following steps: acquiring a data volume relationship of to-be-Hash distributed data corresponding to all associated nodes; and balancing the data volume of the to-be-Hash distributed data corresponding to the associated nodes based on the data volume relationship, so that the data volume of the to-be-Hash distributed data of each associated node is kept within a preset range threshold. According to the scheme of the embodiment, it is guaranteed that the data size corresponding to each association node is kept uniform, so that uniform hash distribution is achieved on the basis of consistent hash distribution, and the situation that some association nodes in one association subsystem are overloaded and some association nodes are overloaded is avoided.

The application discloses an information security data storage system and relates to the technical field of data storage.The application solves the technical problem of avoiding the problem of resource waste or performance bottleneck caused by one-size-fits-all storage by intelligently selecting local storage or distributed storage mode according to the type, capacity size and delay requirement of data.The application dynamically selects local or distributed storage based on data capacity and delay requirement, stores small-capacity and low-delay data locally to reduce network overhead, stores large-capacity and high-delay data in a distributed manner to improve scalability, improves resource utilization, automatically deletes temporary data, combines high-frequency / low-frequency data differentiation processing, takes into account data security and space optimization, dynamically adapts different capacity data through a dynamic sharding strategy, ensures balanced distribution of data through consistent hashing and virtual nodes, and realizes dynamic load adjustment through cold / hot data layering and node performance grading, thereby improving access speed and system stability.

The application provides a state compression consistent hashing method supporting arbitrary removal of an evidence storage node, comprising: constructing a state compression layer for recording a removal state of the storage node; when a storage node is removed, updating the removal state through the state compression layer with constant time complexity; based on the removal state recorded by the state compression layer, addressing a data key, and based on a stateless consistent hashing engine, performing primary mapping on the data key to obtain a first node number; when the storage node corresponding to the first node number is in a working state, outputting the first node number as an addressing result; or when the storage node corresponding to the first node number is in a removal state, triggering secondary mapping, recalculating the first node number in a current working storage node set, and outputting a second node number as an addressing result. The method realizes high efficient search performance and low memory overhead while supporting arbitrary random removal of the storage node.

The invention relates to the technical field of data management, and discloses an AI-based dynamic data management method and system, and the method comprises the steps: obtaining metadata, extracting timestamps, data types and semantic features to generate a metadata set, screening strong correlation features to form a refined feature set, and classifying and determining a normal label set; counting the flow increment in the normal label set at a preset time interval, and when the flow increment exceeds a threshold value, sorting the refined feature set as to-be-distributed messages according to importance to obtain a priority sequence; and constructing a network topology according to the priority sequence, calculating a load balancing factor in combination with a shortest path resource occupancy rate, further calculating a message weight of a message to be distributed and sorting to generate a message distribution queue, and finally distributing the message to a processing node by adopting a consistent Hash algorithm to generate a message distribution result table. According to the method, the message distribution result table is generated through the technical means of screening strong correlation features, counting the flow increment in real time, calculating the message weight and the like, and real-time response under a complex dynamic data management scene is achieved.

The invention discloses a distributed complex task parallel processing system and method. The system comprises a global task scheduling and coordination module, a task division and distribution module, distributed parallel computing nodes, a result aggregation module and a strong consistency verification and fault tolerance module. According to the system, dynamic fragmentation and node mapping of tasks are achieved by calculating task feature fingerprints and adopting a consistent Hash algorithm, and only data migration in the minimum range is needed during node capacity expansion and contraction; by setting a blocking fence synchronization point and a periodic verification mechanism, high consistency and reproducibility of a global task result are realized. And when node faults or inconsistent results are detected, the system automatically rolls back to the last successful fence check point and performs task rescheduling, so that the completeness and stability of calculation are ensured. The method has remarkable advantages in concurrency performance, consistency guarantee and fault-tolerant recovery, and is suitable for scenes such as large-scale data analysis, scientific simulation and real-time decision calculation.

The application provides a stock selection system based on a distributed efficient index formula, comprising a front-end agent service module, a plurality of sharding calculation service modules and a data source service module. The front-end agent service module receives a user request containing a plate factor and an index formula, acquires a benchmark stock pool list requiring stock selection according to the plate factor, divides the benchmark stock pool list requiring stock selection into a plurality of shards by taking the modulus of an incremental internal code and the total number of shards, and generates a corresponding shard calculation request; a shard calculation node management module in the front-end agent service module distributes the request to the corresponding shard calculation service module according to the shard identifier by using an adaptive consistent hashing algorithm. The application separates the front-end agent and the shard calculation by combining the MapReduce task distribution and the result aggregation mechanism, improves the real-time performance and the concurrent processing capability of stock selection calculation, dynamically adjusts the distribution of virtual nodes and entity nodes by using the adaptive consistent hashing strategy, solves the problem of unbalanced sharding, and supports node elastic scaling.

This application relates to the technical field of data processing, and discloses a processing method and system for improving the efficiency of cross-system data consistency comparison, including: performing a data alignment process, including generating a data fingerprint for each data based on consistent hashing; real-time monitoring of data change events, and issuing an analysis and comparison command when a data change event is detected; performing a cross-system data comparison process when the analysis and comparison command is received, and when the comparison result is consistent, the data synchronization is completed, otherwise an alarm is triggered and a repair strategy is executed; adjusting the comparison strategy of the cross-system data comparison process according to the comparison result feedback. The present invention can adjust the cross-system data comparison process through the comparison result feedback. This stepped comparison method can greatly improve the efficiency of the comparison process. At the same time, when there is a data volume inconsistency, the comparison strategy adopted in the comparison process can be adjusted in a timely manner, thereby improving the accuracy of the comparison.