Prioritized Hashed Key for Event Stream Prioritization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage techniques lack effective built-in support for event prioritization in streaming data platforms, leading to poor control over event prioritization, errors, and inefficiencies, particularly due to the lack of real-time prioritization and flexible resource allocation.
Innovation Solution
The implementation of an ordered event stream (OES) system that uses a prioritized hashed key (PHK) to facilitate real-time event prioritization by distributing events across multiple segments based on their priority, allowing for efficient scaling and flexible resource allocation, enabling events to be read and written in a priority-driven manner within the OES system rather than relying on external applications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple independent streams are created for event prioritization, then event priority control is improved, but system complexity increases and real-time processing capability deteriorates
Solution Approach 1:
The patent merges multiple priority streams into a single unified stream that maintains priority information through metadata tags. Instead of creating separate physical streams for each priority level, the system combines all events into one stream while embedding priority indicators (e.g., priority fields, timestamps, or metadata) that allow consumers to filter and process events according to their priority levels. This reduces the number of stream components from N (one per priority) to 1, while preserving priority-based processing capability.
Solution Approach 2:
The unified stream structure serves multiple functions simultaneously: it acts as a single storage container for all events regardless of priority, provides a common interface for stream consumers, enables flexible priority filtering through metadata, and supports both time-ordered and priority-ordered processing modes. This multi-functional design eliminates the need for separate specialized streams for each priority level.
2Adaptability or versatility
If application-level prioritization is implemented, then flexibility in event processing is improved, but control over event prioritization by the storage system deteriorates
Solution Approach 1:
The system introduces an intermediary layer between the storage system and application consumers that handles priority management. The storage system writes all events to a unified stream with embedded priority metadata, and an intermediary mechanism (such as a priority-aware consumer or filtering layer) processes these events according to priority levels. This intermediary approach allows the storage system to maintain full control over event ingestion and prioritization while still providing flexible processing capabilities to applications.
Solution Approach 2:
Events carry their own priority information as metadata within the stream record, enabling them to be self-identified for priority-based processing. The priority information is embedded in the event structure itself (e.g., as a priority field, timestamp, or metadata tag), allowing consumers to autonomously filter and process events based on their priority without requiring external control mechanisms. This self-service approach maintains reliability while providing flexibility.
3Adaptability or versatility
If time-shared soft prioritization is used, then resource allocation flexibility is improved, but implementation complexity and real-time performance deteriorate
Solution Approach 1:
The system performs preliminary prioritization action at the time of event ingestion by embedding priority metadata in each event record as it is written to the stream. This advance preparation eliminates the need for complex runtime prioritization logic and time-sharing mechanisms. When consumers need to process events by priority, the sorting and filtering can be performed efficiently using the pre-established metadata, rather than implementing complex time-shared access patterns with multiple streams and proportional reading logic.
Data Source
AI summary
Event prioritization for an ordered event stream (OES) is disclosed. Unlike conventional prioritization techniques, the disclosed subject matter can be performed by an OES data storage system to provide direct, rather than indirect, control of prioritization. In an embodiment, a prioritized hashed key (PHK) can be determined from an event characteristic and an indicated event priority value based on a selectable priority-sensitive hashing function. As such, events with a same key characteristic but different indicated priorities can have different PHKs, events with different key characteristics but the same indicated priority can have different PHKs, and events with the same key characteristic and the same priority can have a same PHK. An event priority can be inherently comprised in the PHK without needing to explicitly store the priority value with a written event in the OES. Moreover, the disclosed prioritization for the OES can be compatible with OES scaling techniques.