Sharded Database Architecture for Scalable Data Distribution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large database systems face challenges in efficiently managing and scaling data storage due to increasing data volumes, requiring solutions that improve performance, scalability, and fault tolerance while maintaining the benefits of relational databases.
Innovation Solution
The implementation of a sharded database architecture that horizontally partitions data across independent database instances, allowing for scalable, fault-tolerant, and performant data management by distributing data across multiple shards, with a shard catalogue for configuration and a shard director for routing and schema propagation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is stored on a single database instance, then data consistency and relational database benefits are maintained, but scalability and performance deteriorate due to significant computing resource requirements
Solution Approach 1:
The patent divides a large database into multiple smaller shards distributed across independent database instances. Each shard contains a subset of the total data, allowing the system to scale horizontally by adding more shards rather than increasing the capacity of a single database instance. This segmentation resolves the contradiction by enabling large-scale data storage while keeping individual database instances manageable in size and resource requirements.
2Productivity
If data is sharded across multiple database instances, then scalability and performance are improved, but system complexity and data distribution management worsen
Solution Approach 1:
The patent introduces a catalog database as an intermediary that stores metadata about all shards including their locations, data ranges, and distribution patterns. This catalog acts as a central registry that simplifies shard management by providing a single point of control for tracking and managing distributed data across multiple database instances, thereby reducing the complexity of managing sharded systems.
Solution Approach 2:
The catalog database serves multiple functions: it stores metadata about shards, routes queries to appropriate shards, manages data distribution policies, and provides a unified interface for accessing distributed data. This multi-functionality consolidates various management tasks into a single system component, reducing overall system complexity while maintaining scalability.
3Productivity
If non-relational databases are used for large data storage, then scalability is improved, but data consistency, atomicity, and relational query capabilities deteriorate
Solution Approach 1:
By segmenting data into relational shards that maintain ACID properties within each shard, the system achieves scalability while preserving data consistency. Each shard operates as an independent relational database instance that guarantees atomicity and consistency for its portion of the data, while the overall system scales by distributing these reliable shards across multiple instances.
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
Techniques are provided for creating, organizing, and maintaining a sharded database. A sharded database can be created using user-defined sharding, system-managed sharding, or composite sharding. The sharded database is implemented with relational database techniques. The techniques described provide improvements to load distribution, organization, query processing, and schema propagation in a sharded database.