Multi-level caching event lookup for distributed systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing log joining systems face challenges in scaling effectively for continuous event streams due to high latency during lookups, as they can only search within an interval window and complexity increases with larger systems, making them inefficient for real-time event correlation.
Innovation Solution
A distributed event joining system that utilizes a cache layer for short-term storage, processing event information in timestamp order, where the last event in the series is written first, and overwriting the oldest timestamp, with the option to send data to long-term storage, allowing for efficient lookup of primary keys and reducing reliance on disk-based lookups.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a distributed event joining system uses a cache layer for short-term storage to reduce lookup latency, then lookup speed improves, but system complexity increases
Solution Approach 1:
The system divides the storage architecture into multiple layers: a cache layer for short-term storage of recent events and long-term storage for historical data. This segmentation allows the system to handle different types of lookups efficiently while maintaining manageable complexity through clear separation of concerns between layers.
Solution Approach 2:
The cache layer acts as an intermediary between the event stream and the long-term storage system. By introducing this intermediate layer, the system reduces direct access to complex long-term storage, thereby improving lookup speed for recent events while isolating the complexity of long-term storage management.
2Productivity
If the system processes event information in timestamp order with last event written first, then lookup efficiency improves, but data organization complexity increases
Solution Approach 1:
The system inverts the traditional write order by processing and writing the last event in the series first, rather than writing events in chronological order from first to last. This inversion optimizes lookup efficiency by ensuring that the most recent events are immediately available in the cache layer, while the complexity of maintaining this reverse order is managed through structured data organization.
3Adaptability or versatility
If the cache layer handles continuous event streams with low latency lookups, then system scalability improves, but processing load increases
Solution Approach 1:
The cache layer handles only the most recent events that are likely to be queried, rather than processing and storing all events indefinitely. This partial action approach allows the system to scale efficiently by focusing processing resources on the subset of data that provides the most value, reducing overall processing load while maintaining scalability.
Solution Approach 2:
The system changes the temporal parameter of data retention by implementing a time-based expiration policy in the cache layer. Events are retained only for a limited short-term period, which reduces the processing load on the cache while allowing the system to scale by periodically expiring old data and making room for new events.
Data Source
AI summary
Aspects of the disclosure relate generally to a distributed event joining system for joining primary and secondary events based on a common key. In one example, a continuous stream of primary events may be submitted by a plurality of ad serving computers for storage at a cache layer. Each primary event may be associated with a primary event key. After a time T, the stream of primary events may be sent to long term memory. The system may also receive secondary events, each associated with a primary key. The primary key may be used to identify a primary event in the cache layer. If the primary event is not available in the cache layer, the long term memory may be searched for the primary key. Paired events may then be submitted to a distributed joined event storage system.