Ordered Event Stream Scaling via Writer Group Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage techniques for ordered event streams face challenges in scaling while maintaining event order, limiting the number of reader instances and inefficiently utilizing computing resources, especially when event loads exceed threshold levels.
Innovation Solution
The proposed solution involves dynamically scaling the event stream by adjusting the number of parallel segments and allocating computing resources based on writer group characteristics, such as event density and writer instance counts, to accommodate changing workloads and optimize resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the number of parallel reader instances is increased to improve reading throughput, then reading productivity is improved, but event order cannot be maintained
Solution Approach 1:
The event stream is divided into multiple parallel segments, where each segment can be read by a separate reader instance. This segmentation allows multiple readers to operate concurrently on different segments while maintaining event order within each segment, thereby improving reading throughput without compromising event order guarantees.
Solution Approach 2:
The patent introduces a new dimension for parallelism by creating parallel segments that can be read simultaneously. Instead of allowing multiple readers on a single stream (which would break order), the system adds a segment dimension, enabling readers to operate in parallel across segments while maintaining order within each segment's event sequence.
2Productivity
If the event stream is scaled to accommodate more writer instances, then writing productivity is improved, but the complexity of managing segment topography increases
Solution Approach 1:
The system dynamically adjusts the number of parallel segments based on the number of active writer instances. When writer instances are added or removed, the segment topography automatically scales to match, optimizing resource utilization and simplifying management by maintaining a direct correspondence between writers and segments rather than requiring static, complex topography management.
Solution Approach 2:
The patent changes the parameter of segment count dynamically based on writer instance count. By making the segment topography a variable parameter that adapts to workload conditions rather than a fixed structure, the system simplifies management while accommodating scaling requirements for multiple writer instances.
3Device complexity
If fixed segment topography is used to simplify management, then device complexity is reduced, but resource utilization becomes inefficient under varying workloads
Solution Approach 1:
The system transitions from static to dynamic segment topography management. The number of parallel segments automatically adjusts based on the number of active writer instances, ensuring optimal resource utilization under varying workloads while maintaining simple management through automated adaptation rather than complex manual configuration.
Data Source
AI summary
Scaling of an ordered event stream (OES) based on a characteristic of one or more writer groups is disclosed. Scaling a portion of an OES contemporaneous to writing events into that portion can conserve computing resources in contrast to more conventional scaling techniques. Moreover, scaling an OES contemporaneously with writing events thereto can enable improved management of OES scaling for applications that can both read events from an input portion of an OES and, via interim events and interim portions of an OES, write events to an output portion of an OES. An application instance can therefore simultaneously act as both a reader group and writer group and can manage data via interim OESs, such that effects of passing the data through the interim OESs can be cascaded into a scaling of the output portion of an OES based on the writer group characteristic.