Real-time Push API for Enterprise Log Event Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Asynchronous log processing in enterprise threat detection systems is inefficient and introduces complexity, leading to incomplete data transfer due to configuration errors and lack of pre-configured log entries, which reduces the effectiveness of threat detection functionality.
Innovation Solution
Implementing a real-time push API that synchronously sends log entries from applications to the enterprise threat detection system, using a streaming component and runtime parser to map and enrich data, eliminating the need for complex configurations and ensuring all log entries are transmitted without dependency on application-side settings.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If asynchronous log processing is used, then system complexity is reduced, but processing efficiency and real-time detection capability deteriorate
Solution Approach 1:
The patent inverts the traditional asynchronous pull model by implementing a synchronous push model where log sources actively send logs to the ETD system in real-time. This inversion resolves the contradiction by achieving high processing efficiency through synchronous real-time transmission while keeping the ETD system's complexity manageable through standardized API interfaces and centralized configuration.
2Ease of operation
If application-side log configuration is required, then log transmission control is improved, but reliability deteriorates due to configuration errors
Solution Approach 1:
The patent introduces an intermediary real-time push API layer between log sources and the ETD system. This intermediary handles the configuration and translation of log data, allowing log sources to send logs with minimal configuration while the API ensures proper formatting and delivery. This resolves the contradiction by maintaining operational simplicity while improving reliability through centralized validation and error handling.
Solution Approach 2:
The system enables self-service log transmission where log sources automatically send logs through the real-time push API without requiring complex pre-configuration. The API itself provides the configuration management and validation, allowing log sources to simply call the API with basic parameters while the system handles the complex configuration requirements automatically.
3Use of energy by moving object
If asynchronous periodic reading is used, then system resource consumption is reduced, but data timeliness deteriorates
Solution Approach 1:
The patent implements continuous real-time log transmission through the push API, eliminating the periodic gaps inherent in asynchronous reading. Log sources continuously send logs as they are generated, ensuring no data timeliness loss. The system manages resource consumption efficiently by processing logs in real-time streams rather than requiring periodic batch processing, maintaining both continuity and resource efficiency.
Data Source
AI summary
A log entry is received at a streaming component of an enterprise threat detection (ETD) system from a real-time push application programming interface (API) associated with a backend computing system. The received log entry is parsed using a runtime parser associated with the streaming component into mapped data in an ETD format compatible with the ETD system. The mapped data is transferred to an ETD streaming project and enriched. The streaming component writes the enriched data into a database associated with the ETD system.