Sub-routing Key Splits for Distributed Streaming Data Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing streaming data storage systems face challenges in managing uneven ingestion traffic, particularly when a large number of events have the same routing key, leading to performance bottlenecks despite scaling mechanisms, as they cannot effectively distribute the load across different segment stores.
Innovation Solution
The solution involves subdividing a data stream's segment into a segment group, where sub-segments are created and managed internally to preserve ordering and consistency, allowing for even distribution of traffic with the same routing key, and using writer identifiers to maintain ordering guarantees across sub-segments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If segment scaling is performed to handle high ingestion rates, then the system can distribute load across multiple segments, but it cannot effectively distribute traffic with the same routing key across different segment stores
Solution Approach 1:
The patent introduces segment groups that divide a single segment into multiple sub-segments (e.g., 5 sub-segments) while maintaining the same routing key space. This internal segmentation allows traffic with the same routing key to be distributed across multiple sub-segments within the segment group, effectively resolving the load distribution problem for high-volume routing keys while preserving the benefits of segment scaling for overall system capacity.
2Ease of operation
If multiple sub-segments are created within a segment group, then traffic distribution is improved, but maintaining ordering and consistency becomes more complex
Solution Approach 1:
The patent introduces a coordinator component that acts as an intermediary to manage ordering and consistency across sub-segments. The coordinator assigns sequence numbers to events and tracks the state of each sub-segment, ensuring that events are processed in the correct order across all sub-segments. This intermediary mechanism simplifies the complexity of maintaining ordering by centralizing the coordination logic.
Solution Approach 2:
The patent changes the parameter of segment identification by introducing segment group IDs and sub-segment indices as additional layers of organization. Instead of using a single segment identifier, the system now uses a hierarchical structure (segment group ID + sub-segment index) that enables both distributed storage and coordinated management. This parameter change allows the system to track and manage ordering across multiple sub-segments while maintaining a clear hierarchical relationship.
3Loss of energy
If segment merges are performed to consolidate low-traffic segments, then resource utilization is improved, but performance bottlenecks persist for high-volume routing keys
Solution Approach 1:
The patent introduces dynamic segment groups that can automatically adjust their internal structure based on traffic patterns. When a routing key experiences high volume, the system dynamically creates or expands segment groups for that routing key, dividing the load across multiple sub-segments. When traffic decreases, the system can consolidate back to fewer segments. This dynamic adaptation allows the system to optimize for both resource utilization and handling capacity based on real-time conditions.
Data Source
AI summary
A streaming data storage system provides for segment groups that facilitate the distribution of event traffic among storage entities. When too much event traffic with the same routing key (or with a small number of routing keys) is being ingested by the system into one segment of a data stream, the system subdivides the segment into a segment group of sub-segments, which can be distributed among storage resources of the streaming data storage system. Writer components of the system append events received from upstream applications to the sub-segments along with a writer identifier and a sequence number for each event. Reader components of the system use writer identifiers and the sequence numbers to serve events from the segment group to upstream code in the proper order, on a per-writer basis.