Streaming Anchor Journaling via Persistent Index
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing stream processing systems face challenges in efficiently recovering and processing streaming data anchors, particularly in reducing memory load and ensuring effective recovery of computing nodes.
Innovation Solution
The system employs a persistent indexed data structure to store property values associated with streaming data, allowing anchors to include indices into this structure. This approach reduces memory load and facilitates efficient recovery by allowing downstream nodes to retrieve necessary data using anchor indices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full anchors are stored and transmitted in stream processing systems, then downstream nodes can accurately track data points and ensure exactly-once processing, but memory consumption increases significantly
Solution Approach 1:
The patent extracts only the essential identifying information from full anchors and stores it in a compact index data structure. Instead of transmitting complete anchor objects, the system transmits compact index entries that reference the pre-stored anchor information, significantly reducing memory consumption while maintaining the ability to uniquely identify and track data points across the stream processing system
Solution Approach 2:
The patent creates a simplified copy of the anchor information in the form of an index entry. The index stores a condensed representation (hash or abbreviated form) of the full anchor, which serves as a reference key. This copied index structure maintains the unique identification capability of full anchors while occupying minimal memory space, allowing the system to achieve exactly-once processing with reduced resource requirements
2Reliability
If anchor information is persisted for recovery purposes, then system reliability during node failures is improved, but the complexity of the data structure and storage requirements increase
Solution Approach 1:
The patent segments anchor management into two distinct components: a persistent index data structure that stores compact anchor references, and the full anchor objects that are generated and transmitted as needed. The index serves as a lightweight persistent layer that enables recovery without requiring the full complexity of complete anchor objects to be persisted, thereby reducing data structure complexity while maintaining recovery capability
Solution Approach 2:
The patent introduces an index data structure as an intermediary between the full anchor objects and the persistence layer. This index acts as a mediator that stores only the essential identifying information needed for recovery, eliminating the need to persist complete anchor objects. The intermediary index simplifies the persistence mechanism while ensuring that full anchors can be reconstructed or retrieved when needed for system recovery
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Described herein is a system and method of journaling of a streaming anchor resource. An input node can store a value of a property associated with the streaming data in a persistent indexed data structure. The input node can generate an anchor that describes a particular point in time in a data stream. The anchor can include an index into the persistent indexed data structure of the stored value of the property associated with the streaming data. The generated anchor and streaming data can be provided to the downstream node. During recovery of a downstream node, the input node can utilize a received anchor to retrieve a value of a property associated with the streaming data from the persistent indexed data structure, and, provide a batch of data based upon the received anchor and the retrieved property value.