Event Stream Enrichment With Distributed Entity State Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing cybersecurity systems face challenges in efficiently processing arbitrary event streams from diverse sources without relying on hard-coded logic, as they lack the ability to identify and correlate entities and enrich events with state information, necessitating expensive database queries.
Innovation Solution
A cybersecurity management system that uses a set of configurations to identify entities in event streams, route events to associated nodes for state information tracking and enrichment, eliminating the need for external database lookups.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If external databases are queried to obtain state information for every event, then complete entity state information is available, but processing time and computational cost increase significantly
Solution Approach 1:
The system performs preliminary actions by maintaining entity state information in memory before events require it. Nodes continuously track and update entity states (creation, deletion, modifications) in real-time, so when events arrive, the information is already prepared and immediately available without requiring external database queries.
Solution Approach 2:
The patent introduces an intermediary layer (in-memory entity state store at each node) between the event stream and external databases. This intermediary caches entity state information locally, allowing events to be enriched without directly querying external databases for every event, thus reducing processing time while maintaining information completeness.
2Adaptability or versatility
If hard-coded logic is used to identify entities in event streams, then entity identification is straightforward, but the system cannot handle arbitrary event streams from diverse sources
Solution Approach 1:
The system implements a universal entity identification mechanism that works across arbitrary event streams from diverse sources. Each node uses configuration-based entity identification logic that can adapt to different event formats and sources without requiring hard-coded logic for each specific event type, enabling the system to handle multiple event stream formats universally.
Solution Approach 2:
The system changes parameters by using configurable entity identification rules that can be adjusted based on the specific event stream source and format. Rather than fixed hard-coded logic, the identification parameters (such as entity keys, field names, and identification patterns) can be modified to match different event schemas, allowing flexible adaptation to diverse event sources.
3Reliability
If state information is stored in external databases, then data persistence is ensured, but joins and lookups are required when downstream consumers need entity states
Solution Approach 1:
The patent merges entity state information directly with events by enriching each event with the current state of referenced entities. Instead of keeping events and state information separate in external databases requiring joins, the system combines them into a single enriched event structure that contains both event data and relevant entity state information, eliminating the need for subsequent lookups and improving threat detection efficiency.
Solution Approach 2:
The system performs preliminary enrichment of events with entity state information before events reach downstream consumers. Nodes enrich events with current entity states (such as creation time, deletion time, and other relevant attributes) in advance, so when downstream consumers receive events, the complete information is already packaged and ready for immediate analysis without requiring additional database lookups.
4Productivity
If entity state information is enriched in real-time without database queries, then processing speed improves, but system complexity increases
Solution Approach 1:
The system segments the state tracking functionality into individual nodes that each maintain their own entity state information independently. Rather than a centralized complex state management system, each node separately tracks entities it encounters, dividing the overall complexity into manageable distributed units that can operate autonomously and scale horizontally.
Data Source
AI summary
Techniques for enriching events with entity state data to provide distributing tracking of entity state data are provided. A cyber-security management (CSM) system may provide a set of configurations that each define entity identification information indicating when an entity(s) is referenced by an event being processed. When an event that is part of a stream of events is received, the set of configurations may be used by the CSM to identify an entity referenced by the event. The event may be routed to each node of a set of nodes of the CSM that is associated with the identified entity, where each of the nodes associated with the identified entity may update state information of the identified entity maintained by the node. Each of the nodes associated with the identified entity may also enrich the event with the state information of the entity.