Universal Identifier Generation for Distributed Cluster Nodes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed data storage systems, expanding cluster capacity by adding new servers is challenging, especially in object-oriented databases where data objects must be rehashed and identified by stable object identifiers, leading to complex data redistribution.
Innovation Solution
A method for managing data storage servers involves organizing back-end servers into nodes with unique identifiers, generating universal identifiers combining node and server identifiers, and splitting nodes to balance load, allowing for seamless addition of new nodes without redistributing data, using pseudo-random values and binary tree structures for load balancing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a new server is added to the cluster to expand capacity, then the cluster capacity is increased, but data redistribution and rehashing are required which increases system complexity
Solution Approach 1:
The system segments the identifier space into non-overlapping subsets, with each node assigned a unique subset of ultimate identifiers. This segmentation allows new nodes to be added by allocating them new subsets of identifiers without requiring redistribution of existing data, as each data object's identifier permanently indicates its home node.
Solution Approach 2:
The system performs preliminary action by pre-assigning non-overlapping subsets of ultimate identifiers to each node during cluster formation. This preliminary assignment of identifier subsets to nodes eliminates the need for data redistribution when nodes are added later, as the identifier structure already encodes the correct node assignment.
2Productivity
If data objects are rehashed to move them to new servers, then data is redistributed to balance load, but object identifiers must remain stable which complicates the rehashing process
Solution Approach 1:
Instead of rehashing data objects when adding nodes (the conventional approach), the system inverts the approach by having nodes claim specific subsets of the identifier space. Data objects are then automatically routed to the correct node based on their stable identifiers without requiring rehashing, as the identifier structure inherently encodes the destination node.
3Adaptability or versatility
If nodes are organized with unique identifiers and identifier subsets are assigned to each node, then adding new nodes becomes simpler, but the initial cluster setup and identifier management becomes more complex
Solution Approach 1:
The system creates a universal identifier structure that serves multiple functions: it uniquely identifies data objects, encodes the home node information, and enables automatic data routing. This universal identifier approach simplifies cluster scalability while managing complexity through a unified identifier system that handles multiple purposes simultaneously.
Data Source
AI summary
Methods and apparatus, including computer program products, for managing a cluster of servers organized into nodes. A method of one aspect includes establishing a cluster; establishing a set of ultimate identifiers for nodes resulting from splitting in the cluster; and storing every new data object on a node that has a node identifier that identifies a subset of the set of ultimate identifiers, and providing for the object a universal identifier that combines (i) an object identifier that is unique on the node and (ii) a server identifier that is one of the ultimate identifiers in the subset. A method of another aspect includes generating for a new data object a universal identifier that has a node identifier part that uniquely identifies a node, a reserve part generated at least in part as a pseudo-random value, and an object identifier part that uniquely identifies the object in the node.