Virtual Shards for Database Hotspot Contention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database systems with sharding face performance bottlenecks due to contention and growth limitations, as shards cannot split once established, leading to inefficient data management and query performance.
Innovation Solution
The introduction of virtual shards, which are dynamically generated based on context characteristics and contention levels, allowing for the duplication of hotspot shards and adjusting the number of shards to manage data ingestion and query load, thereby alleviating bottlenecks and improving performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If sharding is used to distribute data across multiple servers, then system capacity and throughput are improved, but performance bottlenecks occur when a single shard becomes a hotspot and cannot split
Solution Approach 1:
The patent segments a physical shard into multiple virtual shards dynamically. When a physical shard becomes a hotspot with high contention, the system creates multiple virtual shards that split the data and requests across them, effectively segmenting the load and eliminating the bottleneck while maintaining the original physical shard structure
Solution Approach 2:
The patent introduces dynamic shard creation and management. Virtual shards are created on-demand when hotspots are detected, and the system dynamically routes requests to appropriate virtual shards. This dynamic approach allows the system to adapt to changing data distribution patterns and contention levels without requiring manual intervention or predefined shard splits
2Productivity
If vertical scaling is used to increase single server capacity, then system performance is improved, but system growth is limited and cannot handle continuous data growth
Solution Approach 1:
The patent enables dynamic horizontal scaling by creating virtual shards on-demand. When data growth causes a physical shard to become overloaded, the system dynamically creates multiple virtual shards to distribute the load, allowing continuous system growth without requiring predefined horizontal scaling configurations or manual intervention
3Ease of manufacture
If shards are created based on fixed shard keys, then data distribution is simplified, but the system cannot adapt when data distribution patterns change and hotspots emerge
Solution Approach 1:
The patent segments the rigid fixed shard key approach into a flexible virtual shard layer. While physical shards remain organized by fixed shard keys for simplicity, the system creates virtual shards that can dynamically segment data based on detected hotspots and changing distribution patterns, allowing adaptation without complicating the physical shard setup
Solution Approach 2:
The patent introduces virtual shards as an intermediary layer between the physical shards and the data requests. This intermediary layer absorbs the complexity of adapting to changing data patterns and hotspot conditions, while the physical shards continue to use simple fixed shard keys. The virtual shards handle the complexity of dynamic routing and load balancing
Data Source
AI summary
A computer-implemented method that includes receiving an ingestion request to ingest data to a database comprising physical shards and detecting that the ingestion request is directed to a first hotspot shard. The first hotspot shard has a contention level over a threshold value. The method also detects context characteristics within the data and generates a first virtual shard based on a first virtual shard key selected from the detected context characteristics. The first virtual shard virtually duplicates at least a portion of the first hotspot shard. The method also includes ingesting the data to the first virtual shard.