Dynamic Shard Data Redistribution for Search Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems experience high search latency when data size in shards approaches storage capacity, leading to inefficient query response times due to the need to manage large data sets in sharded databases.
Innovation Solution
A method that dynamically transfers data from a full shard to another when the data size exceeds a threshold, utilizing historical activity data to optimize transfer times when the user is inactive, thereby reducing latency by balancing data distribution across shards.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If data is stored in a single shard, then data storage is simple, but search latency increases as data size approaches storage capacity
Solution Approach 1:
The patent divides the dataset into multiple shards (first shard, second shard, third shard, fourth shard) and dynamically redistributes data between them based on size thresholds. This segmentation allows the system to maintain fast search performance by preventing any single shard from becoming too large, while the sharding mechanism itself is implemented through database tables and standard SQL operations, keeping the overall complexity manageable.
2Productivity
If data is transferred between shards during active user sessions, then data distribution is optimized, but user experience deteriorates due to interrupted queries
Solution Approach 1:
The system performs data transfers between shards during periods when user activity indicates low demand (such as overnight hours or weekends), rather than during active user sessions. This preliminary action during off-peak times allows the system to optimize data distribution without interrupting ongoing queries or degrading user experience during critical periods.
Solution Approach 2:
The patent implements dynamic data redistribution based on real-time monitoring of shard sizes and user activity patterns. The system adjusts transfer timing and data selection dynamically according to current system state, using thresholds and activity data to determine when and what to transfer, thereby adapting to changing user behavior patterns while maintaining optimal performance.
3Productivity
If the system monitors and responds to shard size thresholds, then search performance is optimized, but system complexity increases due to additional monitoring and transfer logic
Solution Approach 1:
The system continuously monitors shard data sizes and user activity patterns, using this feedback to trigger data transfers when thresholds are exceeded. The monitoring mechanism queries database tables to assess shard contents, and the feedback loop automatically initiates transfers based on predefined conditions, creating a self-regulating system that optimizes performance without requiring complex manual intervention.
Data Source
AI summary
Systems, apparatuses, methods, and computer program products are provided. For example, a computer-implemented method may include receiving one or more datasets from a user. In some embodiments, computer-implemented method may include storing at least a portion of the one or more datasets in a first shard during a first time period. In some embodiments, the first shard comprises shard data. In some embodiments, the shard data is associated with a data size. In some embodiments, computer-implemented method may include determining that the data size of the shard data in the first shard is greater than a first data size threshold. In some embodiments, computer-implemented method may include in response to the determination that the data size of the shard data in the first shard is greater than the first data size threshold, transferring at least a portion of the shard data in the first shard to a second shard.