Dynamic Hash-Space Allocation for Low-Contention Microservice Polling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Microservices architecture in distributed systems face inefficiencies in data polling due to redundant work, heavy transaction contention, and load imbalance, leading to performance bottlenecks and reduced scalability.
Innovation Solution
Implement dynamic data partitioning by defining a hash space, deploying microservices into clusters, electing a leader, and reallocating subsets of the hash space based on heartbeats to ensure consistent event publication and efficient data polling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional polling is used where each microservice polls the entire database, then data consistency can be maintained, but it leads to redundant work and heavy transaction contention
Solution Approach 1:
The patent divides the database into multiple partitions, with each partition assigned to specific microservice instances. This segmentation eliminates redundant polling by ensuring that each microservice only queries its assigned partitions, thereby maintaining data consistency while significantly improving polling efficiency and reducing transaction contention.
2Reliability
If a single process is allowed to access the database with locking, then transaction contention is reduced, but scalability is limited as the number of microservices grows
Solution Approach 1:
By segmenting the database into partitions and assigning them to different microservice instances, the system allows multiple processes to access the database concurrently without contention. Each instance operates independently on its assigned partitions, enabling the system to scale horizontally as more microservices are added while maintaining transaction consistency.
Solution Approach 2:
The patent implements dynamic partition assignment where database partitions are automatically allocated and reassigned based on the current state of microservice instances. This dynamic approach enables the system to adapt to changing loads and instance availability, improving both scalability and resource utilization while maintaining consistency through coordinated partition management.
3Reliability
If microservice instances are unable to acquire locks due to contention, then data consistency is maintained, but load imbalance occurs and cost efficiency is reduced
Solution Approach 1:
The partitioning scheme ensures that each microservice instance has dedicated database partitions to query, eliminating lock contention entirely. This segmentation allows all instances to operate in parallel without waiting for locks, achieving both data consistency through controlled access patterns and cost efficiency by maximizing the utilization of all available instances.
Solution Approach 2:
The system dynamically adjusts partition assignments based on instance health, load, and performance metrics. By changing the allocation parameters in response to system state, the patent optimizes load distribution across instances, preventing underutilization and improving cost efficiency while maintaining consistency through coordinated partition management.
Data Source
AI summary
A method and a system for utilizing dynamic data partitioning to ensure consistency of event publications with associated business transactions in a distributed microservices architecture in order to optimize efficiency in polling data are provided. The method includes: defining a hash space that includes a range of assignable hash values; deploying a respective instance of each microservice to form a cluster of microservices within the distributed microservices architecture; allocating a respective subset of the hash space to each microservice; and facilitating a data polling capability with respect to a data table based on the allocated respective subset of the hash space and the deployed respective instance for each microservice.