Adaptive Shard Key Generation for NoSQL Congestion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
NoSQL databases face congestion issues due to uneven data distribution and high insert/update/delete (IUD) activity, leading to performance bottlenecks and latency, as existing methods fail to adaptively manage shard keys effectively.
Innovation Solution
A method is implemented to collect and analyze traffic patterns in NoSQL databases, generating temporary shards with adaptive keys and indexes when IUD activity exceeds a threshold, redirecting intensive traffic to these shards to ensure even data distribution and efficient performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is distributed across multiple shards in NoSQL database, then query load is distributed and parallel execution increases, but congestion occurs on single shard due to uneven distribution or frequent access to same key
Solution Approach 1:
The patent implements dynamic shard key generation that adapts to changing traffic patterns. The system continuously monitors access patterns and automatically adjusts shard keys to distribute load evenly, preventing congestion while maintaining parallel execution capabilities. This dynamic adaptation resolves the contradiction by making the shard distribution flexible rather than static.
Solution Approach 2:
The system changes the shard key parameter based on observed traffic patterns and access frequencies. By modifying the shard key assignment dynamically, the system redistributes data across shards to prevent hotspots, thereby maintaining balanced load distribution and avoiding congestion while preserving parallel query execution efficiency.
2Reliability
If adaptive shard keys are generated to distribute data evenly, then performance during queries improves, but system complexity increases due to dynamic key management
Solution Approach 1:
The patent implements a self-service mechanism where the system automatically monitors its own traffic patterns and generates adaptive shard keys without external intervention. The database system self-adjusts to optimal distribution patterns, reducing the need for manual configuration and complex external management systems, thereby improving query performance while controlling complexity.
Solution Approach 2:
The system incorporates feedback loops that continuously monitor query performance and access patterns, using this information to dynamically adjust shard key assignments. This automated feedback mechanism enables the system to maintain optimal performance automatically, reducing the need for complex manual management while ensuring reliable query execution.
Data Source
AI summary
Method, computer program product, and computer system are provided. Traffic is collected against a NoSQL database by an activity collector. A database transaction log is periodically extracted and analyzed. The collected traffic and the analyzed database transaction log are input to building a knowledge base of database access patterns. Current traffic is captured and used to compute an activity threshold. Traffic is directed to a workload processor based on the activity threshold. Traffic is directed to an intensive insert/update/delete (IUD) processor in response to the activity threshold exceeding a configured threshold. A plurality of temporary shards is generated along with an adaptive key and adaptive index in the plurality of temporary shards. The intensive IUD processor traffic is re-directed to the plurality of temporary shards while the activity threshold exceeds the configured threshold.