72 results about "Consistent hashing" patented technology

The present disclosure discloses a method, a system and a server of removing a distributed caching object. In one embodiment, the method receives a removal request, where the removal request includes an identifier of an object. The method may further apply consistent Hashing to the identifier of the object to obtain a Hash result value of the identifier, locates a corresponding cache server based on the Hash result value and renders the corresponding cache server to be a present cache server. In some embodiments, the method determines whether the present cache server is in an active status and has an active period greater than an expiration period associated with the object. Additionally, in response to determining that the present cache server is in an active status and has an active period greater than the expiration period associated with the object, the method removes the object from the present cache server. By comparing an active period of a located cache server with an expiration period associated with an object, the exemplary embodiments precisely locate a cache server that includes the object to be removed and perform a removal operation, thus saving the other cache servers from wasting resources to perform removal operations and hence improving the overall performance of the distributed cache system.

A distributed storage system has MDSs (metadata servers). Directories of file system namespace are distributed to the MDSs based on hash value of inode number of each directory. Each directory is managed by a master MDS. When a directory grows with a file creation rate greater than a preset split threshold, the master MDS constructs a consistent hashing overlay with one or more slave MDSs and splits directory entries of the directory to the consistent hashing overlay based on hash values of file names under the directory. The number of MDSs in the consistent hashing overlay is calculated based on the file creation rate. When the directory continues growing with a file creation rate that is greater than the preset split threshold, the master MDS adds a slave MDS into the consistent hashing overlay and splits directory entries to the consistent hashing overlay based on hash values of file names.

The invention discloses a file processing method based on a distributed file system, the system, and a client. The method comprises the following steps: the client sends a file writing request to a meta node; the client receives a file identifier, a data block identifier of each data block, the copy quantity of data blocks and a selectivity factor from the meta node; the client performs hashing operation on the file identifier of the file, the data block identifiers of the data blocks and the selectivity factor to obtain a hashing value; the client finds a consistent hashing ring according to the hashing value, obtains the value of a minimal virtual node greater than the hashing value and clockwise selects virtual nodes consistent with the copies in quantity from the virtual node on the consistent hashing ring; and the client sends the data blocks to different storage devices for storage when the selected virtual nodes meet the pre-condition. According to the invention, the occupation of the storage space of the meta node is reduced, the location speed during data writing can be improved, and the load balance of the system is implemented.

A global namespace (GNS) is constructed across heterogeneous network-attached storage (NAS) devices by organizing metadata servers into a cluster layer having cluster nodes and a consistent hashing overlay layer having overlay nodes. The global namespace is distributed based on workload at the cluster and overlay nodes, and a portion of the GNS namespace may be migrated from one layer to the other. Cluster nodes may be added to the cluster layer, overlay nodes may be added to the consistent hashing overlay layer. Further, a cluster node may become an overlay node or an overlay node may become a cluster node. To access the global namespace, a request for namespace metadata managed by a cluster node is serviced by the cluster node, but if the requested namespace metadata have been migrated to the consistent hashing overlay layer, the cluster node will forward the request to the responsible overlay node for servicing.

The invention provides a mass information storage system. The system comprises a storage system, a metadata server, a coordinator and a data distribution and balance module, wherein the storage system has a data storage function; the metadata server is used for receiving and saving metadata from the storage system; the coordinator is used for receiving data which needs to be written by a client into the storage system, organizing the received data to a key value form and sending to the data distribution and balance module, and the coordinator is further used for receiving an inquiry command of the client, instructing a node of the server to implement an inquiry operation, receiving an inquiry result and returning to the client; and the data distribution and balance module is used for distributing the data to all the nodes of the server of the storage system by applying a consistent hashing algorithm.

The invention discloses a multiple-tier distributed cluster system and solves the possible content transfer of common load balance technology that is based on a polling strategy or content hash. The system solves the problems of cluster expansibility, content transfer and the like by adopting tiered load balance technology and combining the polling strategy and consistent hashing algorithm. The technical scheme of the system is as follows: the system comprises a transport tier load balancer at the front end of the system, a URL (Uniform Resource Locator) consistent hashing balancer at the middle tier of the system, and a real server at the rear end of the system, wherein the transport tier load balancer receives request data transmitted by a client side and conducts load balance; requests of the client side are distributed to the URL consistent hashing balancer at the middle tier; the URL consistent hashing balancer receives the request data of the client side distributed by the transport tier load balancer and distributes the same URL to the same real balancer for service through the consistent hashing algorithm; and the real server stores the data.

A distributed data storage method and system based on multilayer consistent hashing are provided. The system includes a plurality of storage nodes for providing data storage and redundancy protection;a plurality of management nodes for maintaining storage node attributes and virtual groups-storage node mapping information; a plurality of monitoring nodes for maintaining storage node state and processing state changes such as storage node addition, deletion, and failure; and one or more clients that provide storage system access points for applications or users. Storage nodes are organized ina tree structure, with each storage node in each layer of the tree being assigned with multiple identities and a hash space maintaining consisteny hash space. Instead of sharing a hash space between all storage nodes, multiple hash spaces in the storage node architecture tree that are consistent across each layer. The system guarantees the reliability and availability of the stored procedure and avoids the problem of load imbalance when accessing the system.

The invention discloses an HTTP (Hyper Text Transfer Protocol) request processing method, device and system. The method comprises the steps that a proxy server receives hypertext transfer protocol (HTTP) requests sent by clients, wherein the HTTP requests carry uniform resource identifiers URLs and user identifications; the proxy server determines a server group corresponding to the URLs, wherein the server group comprises at least two response servers; the proxy server determines target servers from the server group according to a consistent hashing calculation result of the user identification; and the proxy server sends the HTTP requests to the target servers for processing. The problem that in the prior art, HTTP service cannot be stably provided for the clients when the servers are increased or deleted from the Web server cluster can be solved, when the servers are increased or reduced and the whole system serves, for the HTTP requests sent by most of the clients, the Web service is still provided by the appointed servers, and any influence of increasing or reducing the servers is avoided.

The invention discloses a method for constructing a hierarchical catalogue based on consistent hashing data distribution. The method includes the steps: connecting a physical storage server with a storage hardware device to form / data1- / dataX physical storage catalogues through a local file system, and virtually turning the different physical storage catalogues into different logic storage nodes through a storage software server terminal; enabling the logic storage nodes and the physical storage catalogues to be in one-to-one mapping, and adjusting according to needs to realize dynamic migration; taking a logic storage volume as a unit to mount a file system by a storage client, wherein the logic storage volume consists of one or multiple logic storage nodes; and providing a global namespace through an access layer and data distribution of the client to enable physical storage catalogues of the constructed catalogue hierarchy on the physical storage server to be completely consistent with the global namespace of the client. The method has the advantages that uniformity of bottom physical storage and an upper logic structure is achieved to realize a hierarchical catalogue structure; and complete distribution of catalogue hierarchy is realized, single point of failure can be eliminated, and the like.

A set of metadata associated with backup data is obtained. A consistent hash key for the backup data is generated based at least in part on the set of metadata. The backup data is assigned to one of a plurality of deduplication nodes based at least in part on the consistent hash key.

The invention provides a block chain transparent slicing method, a device and a system. The method comprises the following steps: obtaining a plurality of slicing node information serving the block chain; obtaining the corresponding relationship between the slicing node and the slicing according to the slicing node information through a consistent hash algorithm; obtaining the corresponding relationship between the slicing node and the slicing node according to the slicing node information; constructing a ReDiR tree corresponding to the fragments on a DHT network according to the correspondingrelationship between the fragment nodes and the fragments and the fragment nodes; obtaining contract address information of the transaction data, and obtaining the correspondence relationship betweenthe transaction data and the segments through a consistent hashing algorithm according to the contract address information; obtaining a slice node in a corresponding slice according to the corresponding relationship between the transaction data and the slice; sending the transaction data to the slicing node.

A method is provided that uses a consistent hashing technique to dispatch incoming packets in a stable system prior to adding of a node. The method uses a hash table and assigns hash buckets in the table to each network node. A set of fields in each incoming packet is hashed and is used to identify the corresponding hash bucket. The packets are then dispatched to the network nodes based on the nodes' hash buckets. During an observation period, the method identifies the ongoing sessions by creating a bit vector table that is used to identify the old and new sessions during a re-dispatching period. The method uses the consistent hashing method and the probabilistic method dispatch the incoming packets such that each packet that belongs to an old session is dispatched to the same old node that has been processing the other packets of the session.

The invention discloses a multi-blockchain integration control method, a system thereof, an electronic device and a storage medium. According to the method of the invention, a multi-chain architectureis utilized, and transactions are hashed to a pre-determined certain blockchain for execution according to a consistent Hashing rule, thereby improving transaction execution speed of the whole system. A multi-blockchain integration control module generates a new whole-quantity block according to the newly inserted blocks on the plurality of blockchains, and furthermore inserts the generated new whole-quantity block into a whole-quantity chain. The whole-quantity chain can be used for random telescoping of the whole system.

It is provided a distributed processing system comprising a two or more dimensional grid network, on which a virtual ring of a consistent hash is created, for coupling a plurality of nodes to which hash values are assigned, the plurality of nodes including at least a computational resource, and the nodes arranged at positions adjacent on the virtual ring being arranged at positions capable of communication without via other nodes in the grid network.

The invention discloses a distributed storage control method based on a virtual ring load balancing algorithm. By introducing a consistent hashing algorithm, the change of the number of data items caused by increased nodes can be reduced, and the monotonicity can be improved; and by introducing a virtual ring and a historical modulus table, the problem of data movement caused by the change of the number of storage nodes can be avoided. According to the method disclosed by the invention, the impact of the storage medium size on data mapping can be eliminated, and good redundancy characteristics and high scalability of the cloud storage capacity can be achieved on the basis of ensuring load balancing.

The invention provides a back-to-source method and a back-to-source device for an access request. The back-to-source method and the back-to-source device relate to the field of Internet, and solve theproblems of large back-to-source quantity and low resource utilization ratio of a father node scheme in the prior art. The back-to-source method comprises the steps of: receiving an access request ofa user by an edge node, performing consistent hashing operation on an URL (Uniform Resource Locator) carried in the access request and then determining that the URL accesses a corresponding father node server group; and requesting a source file corresponding to the URL from the father node server group. The back-to-source method provided by the invention is suitable for a CDN (Content Delivery Network), and achieves file back-to-source with flexibility, balance, and a high storage disaster tolerance capacity.

A method for scheduling distributed buffer resources is provided, where in the distributed buffer system, each buffer node is distributed on a virtual circle that is based on a consistent hashing algorithm, and the method includes: monitoring a load value of each buffer node in the distributed buffer system; judging whether a load exception exists in the current distributed buffer system according to the load value; and if the load exception exists in the current distributed buffer system, adjusting a layout of the buffer nodes in the distributed buffer system.

The invention discloses a method for improving QUIC protocol request scheduling efficiency. According to the method, a mapping method based on a connection ID and a consistent Hash ring is adopted torealize the dynamical selection of the connection IDs and efficient scheduling of QUIC protocol requests; virtual server nodes are mapped onto the consistent Hash ring, so that the effective control range of the servers is dynamically determined; the selection interval of the connection ID is dynamically determined based on a mapping relation between the effective control range and the connectionID; and the QUIC requests are mapped onto the ring, so that the requests are efficiently scheduled. With the method of the invention adopted, the problem of the failure of QUIC connection smooth migration in a cluster environment which is caused by a condition that a traditional software load balancing request scheduling method fails to adapt to the characteristics of the QUIC protocol requests can be solved, and negative impact on the performance and scalability of a server cluster can be avoided.

The invention relates to the technical field of data filter, in particular to a Hash ring-based distributed data filter method. The method comprises the following steps: S101, receiving information of a distributed de-weighted cluster by a client; S102, client interface data condition request; S103, node reception request; S104, data filter block positioning: positioning data filter blocks of data after hash complementation according to a partition request sent by an RPC; and S105, data returning: carrying out data existence judgement on corresponding data blocks according to filter keys, returning corresponding states and returning the data. After adopting the structure, the method is capable of realizing a multi-tenant function so that the clients can add filters of established types according to the business demands; the lasting backup restoration of the filters can be realized so that the data loss can be avoided; the whole cluster is constructed on the basis of a consistent hash ring, so that the cluster is filtered to realize linear extension; and for the same filter, a plurality of sub-filters can be constructed, so that the false judgement rate can be reduced.

The invention relates to the field of software of distributed storage systems and discloses a node routing method for a large-scale distributed storage system. The node routing method comprises steps as follows: a node ID is automatically generated with a consistent hashing algorithm when a storage node is added to a distributed storage cluster; the storage node performs iterative query in a network, and ten nodes, nearest to the node ID of the storage node, of nodes found in the iterative query process are recorded in a storage node routing table; the storage node routing table is expanded, and a routing tree is obtained; after the node leaves the distributed storage cluster, the storage node deletes the node from the routing tree, if the node leaving the distributed storage cluster is an adjacent node, searching is started once, and if the number of copies of the storage node in the current storage cluster are smaller than 3, information of the storage node is replicated and sent to a substitute node of the adjacent node. The reliability, the usability and the access efficiency of the system are improved, and the system is easy to expand.

The invention discloses a stochastic gradient descent based spectral hashing method in distributed storage. According to the method, the algorithm training time is shortened by utilizing stochastic gradient descent based on a spectral hashing algorithm with semantic consistency, a consistent hashing algorithm based on Cauchy distribution is further proposed, and each data item is compressed into a one-dimensional real-value by utilizing the algorithm. Therefore, the distributed storage can be realized in dynamic network topology by utilizing a consistent hashing thought, and similar data items can be stored in same or similar storage server nodes. The method enables each storage server node to only need to maintain information of a small amount of neighboring nodes; and when the server nodes are added into or exit from a system, only a small amount of related nodes participate in topology maintenance, so that the convergence speed is increased and the storage accuracy is improved.

The invention discloses a partitioning method for processing virtual asset data based on perception of node computing power. The partitioning method includes the steps of processing between machines and processing inside machines. In the step of processing between machines, the weight is added on the basis of the distributed consistent hashing and is determined according to the relational expression Qi=Q*ai / A. In the step of processing inside machines, related attributes of data are set and data with the related attributes are stored in the same machine node, and virtual asset data platforms with same identifier attributes are stored adjacently. Compared with the prior art, the partitioning method with consideration in machine performance and processing related to identifiers and platforms has the advantages of shortening inquiry time greatly and improving inquiry efficiency when applied to storage of mass information of virtual asset.

The invention discloses a method for automatic spreading of a hotspot file. The method is characterized in that hotspot discovery is determined according to two dimensions: the number of requests, and the bandwidth; and as long as a judgment result is positive according to one of the dimensions, discovery of the hotspot file is determined, and load balancing equipment carries out a hotspot spreading action, that is, the load balancing equipment does not execute the consistent hashing strategy when accessing backend cache equipment but randomly selects the cache equipment to be accessed and enables the file to uniformly access each piece of the cache equipment. The method has the advantages of reasonable structure and high convenience in use; and uniform loads on the equipment in a cluster are achieved, so that the loads of the equipment corresponding to the files are enlarged, and the equipment can conveniently provide services at any time.

The invention discloses a server load balancing method based on improved consistent hashing, and the method can achieve the dynamic adjustment of the node load in a server cluster according to the performance of a server. According to the method, the virtual nodes are constructed according to the performance of the nodes in the server cluster and the connection number in one period, so the problemof non-uniform load of a load balancing system based on consistent hash at present is solved. The method comprises the following specific steps: enabling a routing server to periodically maintain thenumber of nodes and performance data of the server cluster, and calculating the number of virtual nodes to be constructed by each node according to the performance ratio of the nodes and the number of the nodes; constructing a hash ring by the cluster nodes and the virtual nodes according to a hash algorithm; enabling the routing server to periodically maintain the connection number of each cluster node; calculating a virtual node number redundancy value according to the difference of connection numbers under the same performance value after each period; and periodically repeating the five steps, updating the number of the virtual nodes, and reconstructing the hash ring.

The invention discloses a distributed cache method and system based on an improved consistent hashing. Included are a plurality of node servers. Each node server comprises a main cache region and a copy cache region. For each node server, the main cache region is used for caching, according to a consistent hashing algorithm, data written by a user or in need of access on the node server; and the copy cache region is used for caching, according to the consistent hashing algorithm, data of the main cache region of the previous node server on a hashing ring constituted by all the node servers. The invention effectively improves the reliability and availability of the system, and can also avoid a waste of resources and reduce the number of times of cached data synchronization.