IoT Partition Management via Candidate Server Election
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current IoT systems face inefficiencies in electing a primary server and maintaining a sustainable load distribution across a large number of partitions, especially when secondary servers are required, leading to potential server overload and unbalanced loads.
Innovation Solution
A decentralized approach is implemented, where a small set of candidate servers is pre-assigned to each partition, using a consistent datastore protocol to elect a primary server and allowing for efficient replacement, with load balancing strategies that consider fault domains and server loads, ensuring high availability and fault tolerance through independent server evaluations and dynamic reconfiguration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If current protocols are used to elect a primary server in IoT systems with hundreds of servers and thousands of partitions, then the system can handle the scale, but the election process becomes inefficient and cannot efficiently elect a primary server
Solution Approach 1:
The system segments the large fleet of servers into smaller groups, with each partition having a pre-assigned subset of candidate servers (e.g., 3-5 candidates) from the total fleet. This segmentation transforms the impossible task of electing a primary from hundreds of servers into a manageable election among a small subset, directly resolving the efficiency contradiction.
Solution Approach 2:
The system performs preliminary action by pre-assigning candidate servers to each partition before the election process begins. The control plane pre-configures which servers are candidates for each partition based on load balancing strategies and fault domain considerations, so that when election is needed, only these pre-selected candidates need to participate, dramatically improving election efficiency.
2Reliability
If load balancing is implemented across hundreds of servers handling thousands of partitions, then sustainable load distribution is desired, but server overload and unbalanced loads occur
Solution Approach 1:
The system implements continuous feedback mechanisms where servers periodically report their load status to the control plane, and the control plane adjusts partition assignments based on this feedback. The load balancing strategy monitors server health, capacity utilization, and fault domain distribution, dynamically reassigning partitions to maintain balanced loads and prevent server overload.
Solution Approach 2:
The load balancing system is dynamic rather than static. The control plane continuously evaluates server conditions and reassigns partitions in real-time based on changing load conditions, server health status, and fault domain requirements. This dynamic adjustment ensures sustainable load distribution even as system conditions change, resolving the contradiction between maintaining reliability and ease of operation.
3Reliability
If secondary servers are maintained for each partition to ensure high availability, then fault tolerance is improved, but system complexity increases
Solution Approach 1:
The system applies local quality by maintaining different roles for different servers within the same partition context. Each partition has one primary server and multiple secondary servers with distinct functions: the primary handles active requests while secondaries stand by for failover. This local differentiation of server roles within partitions provides fault tolerance through clear specialization, managing complexity through structured role assignment rather than uniform configuration.
4Productivity
If partitions are dynamically moved between servers to balance load, then load sustainability is improved, but partition moves increase system complexity and potential failures
Solution Approach 1:
The system applies preliminary anti-action by pre-configuring multiple candidate servers for each partition before any failure or load imbalance occurs. When the primary server becomes unavailable or overloaded, the system can immediately promote a pre-identified secondary server without needing to search or select from the entire fleet, preventing partition orphaning and reducing the complexity of dynamic repartitioning while maintaining load sustainability.
Data Source
AI summary
A system includes a plurality of servers, a control plane to determine a first partition of a plurality of devices and to determine a subset of the plurality of servers to assign as candidate servers for the first partition, and a common data store comprising a first stream and a second stream. The control plane is to store, in the first stream, a first message indicating the first partition, the candidate servers, the second stream, and a first message tag, the candidate servers elect a primary server of the first partition from the candidate servers using the first stream, and the elected primary server inserts read and write updates associated with the plurality of devices of the first partition into the second stream.