Dynamic Shard Mapping for Database Routing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing sharded database management systems (DBMSs) face limitations in data distribution control, scalability, and performance due to random or schema-mapped sharding approaches, which lack flexibility and efficiency, especially when dealing with small sharding key values or frequent changes.
Innovation Solution
Dynamic shard mapping allows for the dynamic definition, evolution, and redefinition of sharding distributions across shards without modifying synchronized schema metadata, using a distributed sharding directory that maps sharding key values to shards, enabling efficient routing of database commands and optimizing data distribution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If schema-mapped sharding is used to control data distribution, then data distribution control is improved, but device complexity and difficulty of operation increase due to requiring schema metadata changes
Solution Approach 1:
The patent segments the sharding configuration from the database schema by introducing a separate sharding directory structure. This allows data distribution control to be managed independently through the sharding directory rather than modifying schema metadata, thereby reducing the complexity of schema management while maintaining flexible data distribution control.
Solution Approach 2:
The sharding directory acts as an intermediary between the database schema and the data distribution mechanism. It mediates the configuration of sharding key values to shards, eliminating the need to modify schema metadata directly while providing full control over data distribution. This intermediary structure simplifies operation by separating configuration from schema.
2Reliability
If serialized schema metadata is used for sharding configuration, then reliability is improved, but loss of time increases due to overhead associated with schema changes
Solution Approach 1:
The patent separates sharding configuration into an independent sharding directory structure that does not require serialized schema metadata changes. This segmentation allows sharding configuration to be updated without the time-consuming overhead of schema serialization and deserialization, while maintaining reliability through the distributed directory structure.
Solution Approach 2:
The sharding directory enables dynamic configuration changes without requiring schema metadata updates. The system can adapt sharding distributions in real-time by modifying the sharding directory entries rather than performing time-consuming schema changes, thereby reducing time loss while maintaining reliability through consistent data routing.
3Ease of operation
If random sharding is used, then ease of operation is improved, but manufacturing precision (data distribution uniformity) deteriorates for small sharding key values
Solution Approach 1:
The patent implements a dynamic sharding approach where the sharding directory can be configured to handle different scenarios. For small sharding key values, the system can use custom mapping rules in the sharding directory to achieve uniform distribution, while maintaining ease of operation through the simplified directory structure that doesn't require schema changes.
Solution Approach 2:
The sharding directory allows changing the sharding parameters and mapping rules dynamically based on the data characteristics. For small sharding key values, custom mapping parameters can be defined in the sharding directory to achieve uniform distribution, while the overall system remains easy to operate through the simplified configuration mechanism.
Data Source
AI summary
Under dynamic shard mapping, applications may dynamically define, evolve, and redefine how a sharded table is sharded across shards according to sharding keys. Dynamic shard mapping is referred to as being dynamic because changes to a sharding directory that defines a sharding distribution scheme are effected without having to change synchronized f metadata. A sharding directory maps sharding key values to shard servers. The changes are effected directly by invoking an API or indirectly by submitting a DML command that includes, for example, a sharding key value that is unmapped by the sharding directory. The sharding directory is distributed among the shards and client computers of a sharded DBMS to facilitate and optimize the routing of database commands across the shards of a DBMS.