Ordered Stream Event Positioning for Out-of-Order Ingestion

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Contemporary data storage systems like PRAVEGA struggle to provide user-specified ordering of events across multiple segments, as they are designed for immutable append-only sequences, which limits flexibility in consuming data in a desired order, especially in scenarios where events are ingested out of order.

Innovation Solution

Incorporating a position value with each event to define its order within an ordered stream, allowing events to be read in a natural order based on user-specified positions, even across multiple segments, while maintaining metadata for minimum and maximum positions within each segment to manage scaling and ordering.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of manufacture

If events are stored in immutable append-only segments with routing key hashing, then writing efficiency and simplicity are improved, but the ability to consume events in user-specified order deteriorates

Engineering Contradiction:

Improvewriting efficiencyVSAvoidordered consumption flexibility

Core Design Contradiction:

Ease of manufactureVSEase of operation

Solution Approach 1:

The system segments the stream into multiple segments based on position ranges. Each segment is assigned a position range metadata, allowing the system to divide and conquer the ordering problem while maintaining append-only properties within each segment. Readers can efficiently locate and consume events in order by querying segments based on position ranges.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces an intermediary indexing layer that maps position ranges to segment identifiers. This intermediary structure enables ordered consumption without requiring events to be physically reordered on storage, maintaining writing simplicity while enabling flexible reading through the position-to-segment mapping mechanism.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If segments are sealed after writing to ensure immutability, then data consistency is improved, but the ability to handle out-of-order events and scaling deteriorates

Engineering Contradiction:

Improvedata consistencyVSAvoidhandling out-of-order events

Core Design Contradiction:

ReliabilityVSAdaptability or versatility

Solution Approach 1:

The system dynamically manages segment states, allowing segments to transition between writable and sealed states based on position range assignments. Segments can be reopened and repositioned when scaling occurs, enabling the system to adapt to out-of-order event ingestion while maintaining consistency through controlled sealing operations.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The patent performs preliminary position range assignments to segments before sealing them. This preliminary action ensures that even if events arrive out of order, they can be correctly routed to appropriate segments based on their position values, and segments can be sealed once their position ranges are finalized, maintaining both flexibility and consistency.

Inventive Principle:

Principle #10Preliminary action

3Device complexity

If segments are sealed to maintain immutability, then storage simplicity is improved, but scaling operations and segment repositioning deteriorate

Engineering Contradiction:

Improvestorage simplicityVSAvoidsegment repositioning capability

Core Design Contradiction:

Device complexityVSAdaptability or versatility

Solution Approach 1:

The system uses metadata copying and reference mechanisms where segment position range information is maintained independently from the actual event data. During scaling, the system can create new segments with copied position range metadata and redirect readers to appropriate segments without physically moving or unsealing existing event data, maintaining storage simplicity while enabling repositioning.

Inventive Principle:

Principle #26Copying

4Productivity

If events are appended to the tail of segments, then writing speed is improved, but the ability to insert events at arbitrary positions deteriorates

Engineering Contradiction:

Improvewriting speedVSAvoidarbitrary position insertion

Core Design Contradiction:

ProductivityVSAdaptability or versatility

Solution Approach 1:

The patent introduces a position dimension orthogonal to the append-only writing model. Events continue to be appended to segment tails for writing speed, but each event is tagged with a position value. The system then uses this position dimension to logically reorder events during reading operations, achieving arbitrary position access without sacrificing writing performance.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

Data Source

PatentUS11347747B1Ordering data stream events in a streaming data storage platform

Publication Date: 2022.05.31 EMC IP HLDG CO LLC

AI summary

A streaming data storage system maintains streamed events based on external ordering (position) data. Each segment of the stream has minimum position maximum position metadata values based on the events' positions therein. When an ordered event is appended to an ordered segment, an index segment maintains offset information that tracks the event's position order relative to offsets of other previously appended events. When a segment is scaled (split or merged), the predecessor segment set's maximum position is the basis for the successor segment set's minimum position. An event is written to any predecessor segment or successor segment based on the event's routing key and position with respect to a matching segment's routing key and minimum and maximum positions. A data stream can be truncated based on a specified position. Reads can be for events between a specified range of positions, including a range that spans predecessor and successor segments.