Virtual Partitioning Shared Message Bus Auto-Provisioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In multi-tenant cloud environments, traditional data partitioning methods are inefficient and scalable, as they require manual provisioning of new topics for each customer and can lead to scalability issues due to uneven data distribution across partitions, affecting data locality and resource utilization.
Innovation Solution
The implementation of virtual partitioning of a shared message bus, where data from a particular customer is limited to specific partitions of a shared topic, allowing multiple customers to share a single topic until capacity is reached, with larger customers having their own topics based on needs, and using a weighted stable load balancing algorithm for scheduling virtual partitions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional manual provisioning of new topics for each customer is used, then data isolation is ensured, but device complexity and management overhead increase significantly
Solution Approach 1:
Multiple customer topics are merged into a single shared topic with virtual partitioning. Each customer's data is routed to specific partitions within the shared topic based on a hashing function of the customer ID, ensuring data isolation while reducing the number of topics from N (one per customer) to 1 (shared topic). This eliminates the need for manual provisioning of multiple topics and simplifies management.
Solution Approach 2:
The shared topic is segmented into multiple partitions, with each partition dedicated to specific customers based on their customer IDs. The segmentation is achieved through virtual partitioning where partitions are assigned to customers using a deterministic hashing function, ensuring that each customer's data is isolated to specific partitions while sharing the overall topic infrastructure.
2Productivity
If data is evenly distributed across partitions, then resource utilization improves, but data locality deteriorates due to mixed customer data in each partition
Solution Approach 1:
Each partition within the shared topic has a specific quality characteristic: it is dedicated to serving specific customers (identified by customer ID). This local quality ensures data locality because all data for a given customer is routed to the same partition(s) based on a hashing function, while the overall system maintains even distribution across partitions for balanced resource utilization.
3Device complexity
If multiple customers share a single topic, then device complexity is reduced, but scalability is limited by partition capacity constraints
Solution Approach 1:
The system dynamically assigns customers to partitions based on their customer IDs using a hashing function. When new customers are added or existing customers grow, the system automatically routes their data to appropriate partitions without requiring manual intervention or topic recreation. This dynamic assignment enables the system to scale from small to large customers while maintaining the shared topic structure.
4Reliability
If manual topic provisioning is used for each customer, then data isolation is guaranteed, but loss of time occurs due to manual intervention requirements
Solution Approach 1:
The system provides self-service automatic topic provisioning. When a new customer is added to the system, the shared topic automatically routes their data to the appropriate partition(s) based on their customer ID through a deterministic hashing function. This eliminates the need for manual topic creation and assignment, reducing provisioning time from manual intervention to automatic calculation.
Data Source
AI summary
According to embodiments, a method for virtual partitioning of data includes receiving a data stream comprising a plurality of traces, each trace comprising a plurality of spans from a plurality of users. The method also includes assigning the plurality of traces of the data stream to a plurality of virtual partitions based on each user of the plurality of users, each virtual partition of the plurality of virtual partitions comprising data of a user of the plurality of users. The method also includes scheduling at least a subset of the plurality of virtual partitions to at least one user partition of a shared topic, the at least one user partition comprising data from at least one virtual partition of at least one user of the plurality of users. The method also includes indexing each user partition of the shared topic based on each user and each virtual partition.