Database Sharding by Shard Levels for Seamless Expansion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional distributed database systems face inefficiencies when expanding, as adding more data servers requires data relocation and changes the server distribution topology, making the process slow and resource-intensive.
Innovation Solution
The system organizes database shards by levels, using a data processing engine to assign shard levels to users and maintain corresponding shard functions, allowing seamless identification and processing of user data across changing shard levels without data relocation during expansion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If more data servers are added to a distributed database system, then the system's data storage capacity and processing power are improved, but the complexity of data redistribution and topology changes increases, making the expansion process slow and resource-intensive
Solution Approach 1:
The patent segments the database namespace into multiple shard levels (L0, L1, L2, etc.), where each level represents a different granularity of data distribution. This segmentation allows the system to add servers at any level without requiring global redistribution, as each shard level operates independently with its own hash ring and data placement strategy.
Solution Approach 2:
The patent introduces a vertical dimension to data distribution by organizing shards in a hierarchical structure across multiple levels. Instead of a single flat distribution layer, data can be accessed at different shard levels depending on the access pattern, allowing expansion without disrupting other levels. This multi-dimensional organization enables independent scaling at each level.
2Productivity
If data servers are added to maintain read locality for entities, then data access efficiency is improved, but the data migration process becomes more intensive and time-consuming
Solution Approach 1:
The patent pre-allocates shard slots and maintains hash rings at each shard level in advance, so when servers are added, the system can immediately assign data to new shards using the existing hash function without requiring data migration. The shard metadata and routing information are prepared beforehand, enabling seamless expansion.
Solution Approach 2:
The patent creates copies of the hash ring structure at each shard level, allowing the system to maintain multiple independent data distribution maps. When expanding, the system can copy the existing hash ring configuration to new shard levels or modify individual levels without affecting others, enabling parallel expansion and avoiding global data migration.
3Adaptability or versatility
If the server distribution topology is changed to accommodate more servers, then the system's scalability is improved, but the existing access mechanisms become obsolete and require updates
Solution Approach 1:
The patent designs a universal access mechanism that works across all shard levels. The same high-level API and routing logic can access data at any shard level (L0, L1, L2, etc.), making the access mechanism independent of the underlying topology. This multi-functionality allows the system to scale without requiring changes to access patterns or client code.
Solution Approach 2:
The patent introduces a shard routing layer that acts as an intermediary between clients and physical servers. This routing layer translates client requests into shard-level operations using hash functions, abstracting away the complex topology details. The intermediary handles all topology changes internally while presenting a consistent interface to clients, decoupling access mechanisms from physical distribution.
Data Source
AI summary
A system and a method are disclosed for identifying a particular database shard associated with a given user. Each user is assigned a shard level that corresponds to a set of database shards that were active within a database management system when the user first joined the data management system. When a data request associated with a given user is received, the shard level assigned to the user identifies the database shard in the database management system that stores data associated with the user. The data request is thus processed using the identified database shard.