Distributed Database Attosharding for Scalable Data Distribution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed databases face inefficiencies in scaling due to increased inter-node communication and data redistribution, leading to performance decline and making incremental scaling computationally inefficient and time-consuming.
Innovation Solution
Implementing a distributed database as a set of clusters with attosharding, where the key space is divided into segments proportional to the number of nodes, allowing for weighted random distribution of writes and targeted reads, optimizing both write and read operations by ensuring proportional loads across clusters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If consistent hashing is used to distribute data among nodes, then data distribution is achieved, but scaling becomes computationally inefficient and time-consuming due to data redistribution
Solution Approach 1:
The patent divides the key space into attoshards, where each attoshard is a segment containing keys for multiple cluster segments. This segmentation allows the system to work with smaller, manageable portions of data during scaling operations, reducing the computational overhead and time required for data redistribution when nodes are added or removed from the cluster.
2Adaptability or versatility
If the number of nodes in the cluster changes, then cluster scalability is improved, but inter-node communication increases causing performance decline
Solution Approach 1:
The patent implements weighted random distribution of write operations across clusters, where each cluster receives a proportion of writes based on its size. This local quality approach ensures that larger clusters handle more writes while smaller clusters handle fewer, optimizing the distribution of workload and reducing unnecessary inter-node communication overhead, thereby maintaining performance during scaling operations.
3Stability of the object's composition
If data is moved from original node to new node during scaling, then data distribution is maintained, but the operation becomes computationally inefficient
Solution Approach 1:
The patent pre-computes and stores the mapping between tokens and cluster segments in attoshards before scaling operations occur. This preliminary action allows the system to quickly determine data locations during scaling without performing complex real-time computations, reducing the computational energy required while maintaining data distribution consistency when nodes are added or removed.
Data Source
AI summary
A distributed database receives an instruction to read or write data. The instruction includes a key. The database includes a key space defined by attoshards. An attoshard is a segment of key space having a size, in keys, proportional to a total number of nodes in the database. The attoshard includes keys for cluster segments at predefined positions in the attoshard. Each cluster segment corresponds to one cluster. A node of the database hashes the key to generate a token. The node performs a modulo operation on the token using the total number of nodes in the database to compute a remainder value. The node determines a cluster segment of an attoshard based on a position in the attoshard defined by the remainder value. The node determines a cluster for the instruction based on the cluster segment and executes the instruction at the determined cluster.