Event Pipeline Handling Large Payloads via Segmented Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional event pipeline systems face limitations in handling large and data-rich events due to size constraints on event data structures, which restrict data providers and processing components, especially when batching or joining multiple events.
Innovation Solution
The event pipeline system employs separate notification channels and data stores for event notifications and payloads, allowing for arbitrary-sized event payloads and enabling streaming batch and join processes without requiring the entire payload to be stored in memory, thus overcoming size limitations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If event payloads are stored in memory for processing, then processing speed is improved, but memory usage increases and limits payload size
Solution Approach 1:
The system segments the event data structure into two separate parts: a notification channel containing metadata and a reference, and a separate data store containing the actual payload. This allows the notification to be processed quickly in memory while the payload resides in persistent storage, resolving the contradiction between fast processing and large payload capacity.
Solution Approach 2:
The patent introduces an intermediary reference mechanism (event identifier or pointer) that connects the in-memory notification to the payload in external storage. This intermediary allows the system to work with small in-memory structures while accessing large payloads externally, eliminating the need to load entire payloads into memory for processing.
2Stability of the object's composition
If size constraints are imposed on event data structures, then system stability is improved, but adaptability to large payloads deteriorates
Solution Approach 1:
The system transitions from a single-dimension in-memory data structure to a multi-dimensional architecture where notifications exist in the in-memory space and payloads exist in external storage space. This dimensional separation allows the system to maintain stable, small in-memory structures while adapting to arbitrarily large payloads in external storage.
3Reliability
If entire payloads are loaded into memory for batch operations, then processing completeness is improved, but memory efficiency deteriorates
Solution Approach 1:
The system performs preliminary actions by storing complete payloads in external data stores before processing operations. During batch operations, only the necessary notification metadata and references are loaded into memory, while the actual payload data remains in external storage, allowing complete processing without loading entire payloads into memory.
Data Source
AI summary
Systems and methods are provided for automatically orchestrating the handling of events through a processing pipeline without limitation (or without a substantial limitation) as to the size of the event payload associated with the event. The event pipeline system stores event payloads in data stores and generates notifications regarding the events. The notifications may be placed into event streams for handling by various processing components of the event pipeline system. The processing components may receive notifications or events that they are to process, and may separately access event payloads from the data stores. The processing components may generate and save processed event payloads to the data stores in a streaming fashion such that the computing resources of the processing components do not limit (or substantially limit) the size of the event payloads that the processing components may handle.