Hot Data Migration in Redis Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Redis clusters face service breakdowns due to the 'hot key' phenomenon, where excessive access requests concentrate on a single server, leading to overload and failure, which subsequently breaks the cache mechanism and backend databases.
Innovation Solution
A method for migrating hot data from a primary Redis cluster to a secondary cluster when certain conditions are met, distributing service pressure and ensuring high performance and reliability by using a monitoring process to identify hot data and trigger its migration to a secondary cluster with joint management and updated access paths.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a single-threaded main control unit processes access requests sequentially, then data access reliability is ensured, but service pressure concentrates on one server causing system breakdown
Solution Approach 1:
The patent divides the single Redis cluster into a primary cluster and a secondary cluster. The monitoring process segments access requests by identifying hot data and routing them to the secondary cluster, thereby distributing service pressure while maintaining the single-threaded reliable processing model in each cluster.
Solution Approach 2:
The monitoring process acts as an intermediary between access requests and the Redis clusters. It monitors access patterns, identifies hot data, and triggers migration to the secondary cluster, mediating the load distribution without affecting the reliable sequential processing of individual clusters.
2Productivity
If hot data is migrated to a secondary cluster, then service pressure on the primary cluster is reduced, but system complexity increases due to cluster management
Solution Approach 1:
The monitoring process implements self-service by automatically monitoring access patterns, identifying hot data, and triggering migrations to the secondary cluster without external intervention. This automation reduces operational complexity despite the added architectural complexity of dual clusters.
Solution Approach 2:
The system changes the parameter of data distribution by migrating hot data from the primary cluster to the secondary cluster based on monitored access patterns. This parameter change (data location) resolves the service pressure issue while the monitoring mechanism manages the complexity.
3Speed
If all access requests are centralized on a same physical device according to hash value, then data access speed is improved, but the physical device breaks down due to overload
Solution Approach 1:
The patent segments the centralized physical device into multiple physical devices hosting the primary and secondary clusters. Hot data is routed to the secondary cluster on a different physical device, distributing the load while maintaining fast access speeds through the optimized dual-cluster architecture.
Solution Approach 2:
The system adds a dimensional change by introducing a secondary cluster dimension alongside the primary cluster. This allows access requests to be routed to a different physical dimension (secondary cluster on another device) for hot data, preventing single-device overload while maintaining speed.
Data Source
AI summary
A first cluster and a second cluster include at least one single-threaded main controller, the at least one single-threaded main controller processes one access request requesting for data at a time, and the data accessed by the access request is jointly managed by the first cluster and the second cluster. A database server determines hot data managed by the first cluster and determines whether a migration condition for migrating the hot data to the second cluster is met. The second cluster manages hot data that meets the migration condition. The database server migrates the hot data that meets the migration condition to the second cluster and triggers an update of an access path of the hot data.