Real-time Log Joining via Cache Index Map for Event Streams
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Log joining systems face scalability issues when handling continuous streams of events, as existing batch-based methods do not efficiently manage the joining of primary and secondary events in larger systems.
Innovation Solution
A method and system that utilize a short-term memory cache layer and a long-term memory for storing primary events, generating an index map to quickly determine the presence of primary event keys, allowing for efficient joining of secondary events with either short-term or long-term stored primary events, and storing the joined events in a joined event storage system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If batch-based map-reduce methods are used for log joining, then joining accuracy is maintained, but system scalability and processing speed deteriorate in larger continuous stream systems
Solution Approach 1:
The patent divides the log joining system into two distinct segments: a cache layer for storing recent primary events with an index map for fast lookup, and a long-term storage system (HDFS) for historical events. This segmentation allows the system to handle continuous streams efficiently by routing recent events to the fast cache layer while maintaining scalability by offloading historical data to distributed storage, thereby resolving the contradiction between joining speed and system complexity.
Solution Approach 2:
The system performs preliminary actions by pre-storing primary events in the cache layer with pre-computed index maps before secondary events arrive. When secondary events need joining, the system can immediately query the pre-prepared index map in the cache layer without performing complex map-reduce operations at join time, thus improving processing speed while maintaining manageable system complexity.
2Reliability
If all primary events are stored in long-term memory, then lookup completeness is improved, but access latency and disk load increase
Solution Approach 1:
The patent applies local quality by creating a specialized cache layer with different storage characteristics than long-term memory. The cache layer stores recent primary events with an optimized index map structure that enables O(1) or O(log n) lookup time, while long-term HDFS storage handles historical events. This local optimization ensures that frequent recent lookups are fast while maintaining completeness by falling back to HDFS when needed, resolving the contradiction between lookup completeness and latency.
Solution Approach 2:
The cache layer acts as an intermediary between secondary events and the long-term HDFS storage. Instead of directly querying HDFS for every secondary event, the system first checks the cache layer's index map. This intermediary layer filters out many lookups that can be resolved quickly in memory, reducing the burden on HDFS and minimizing lookup latency while maintaining lookup completeness through the fallback mechanism.
3Productivity
If sequential batch processing is used, then joining accuracy is maintained, but processing throughput decreases in continuous stream systems
Solution Approach 1:
The patent implements continuous processing by maintaining a continuously updated cache layer that receives and indexes primary events as they arrive in the stream. Secondary events are joined in real-time against this continuously maintained index map, eliminating the need to stop and restart processing in batches. This continuous action maintains joining accuracy through consistent index management while dramatically improving processing throughput by avoiding batch processing interruptions.
Solution Approach 2:
The system performs preliminary indexing of primary events in the cache layer as they arrive, maintaining the index map in a ready state before secondary events need joining. This preliminary organization of data in memory with proper indexing structures ensures that when secondary events arrive, they can be joined accurately and quickly without requiring complex batch processing, thus maintaining joining accuracy while improving continuous processing throughput.
Data Source
AI summary
Aspects of the disclosure relate generally to a distributed event joining system for joining primary and secondary events based on a common key. In one example, a continuous stream of primary events may be stored at a cache layer. Each primary event may be associated with a primary event key which may be used to generate an index map for the cache layer. After a time T, the stream of primary events may be sent to long term memory. The system may also receive secondary events, each associated with a primary key. The primary key may be used to identify a primary event in the cache layer. If the primary event is not available in the cache layer, the long term memory may be searched for the primary key. Paired events may then be submitted to a distributed joined event storage system.