Adaptive API Poller Scheduling for Cybersecurity Event Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge of scaling cybersecurity threat detection and response services to handle an increasing volume of threats without causing technical inefficiencies, such as delayed detection and response, is exacerbated by the need to efficiently manage polling tasks across cloud-based services.
Innovation Solution
A computer-implemented method that dynamically computes an optimal polling frequency control value based on historical polling performance data, partitions the time span into sub-intervals, and transmits network requests to retrieve raw event data, thereby optimizing memory and time usage, reducing the number of requests, and accelerating threat detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the polling frequency is increased to detect more security threats, then the detection capability is improved, but the memory consumption and API response latency worsen
Solution Approach 1:
The patent divides the polling task into segmented time intervals and processes data in smaller chunks rather than retrieving all data at once. This segmentation allows the system to maintain high detection frequency while reducing peak memory consumption by processing and discarding data incrementally.
Solution Approach 2:
The system dynamically adjusts polling frequency and data retrieval parameters based on system conditions and threat levels. This dynamic approach allows optimization of memory usage while maintaining adequate detection capability across varying operational states.
2Reliability
If the polling frequency is increased to detect more security threats, then the detection capability is improved, but the API response latency increases
Solution Approach 1:
By segmenting the polling intervals and processing data in smaller batches, the system reduces the time each API request takes to complete. This allows more frequent polling without proportionally increasing total response latency, as each individual request is lighter and faster.
Solution Approach 2:
The system performs preliminary filtering and processing of data before full analysis, allowing rapid initial assessment that reduces overall response time while maintaining comprehensive detection capability.
3Measurement precision
If the data retrieval volume is increased to improve threat detection, then the detection accuracy is improved, but the processing time and system efficiency worsen
Solution Approach 1:
The system extracts and processes only the most relevant security event data from the retrieved information, discarding redundant or low-priority data. This extraction approach maintains high detection accuracy by focusing on critical threats while improving processing efficiency by reducing the volume of data requiring full analysis.
Solution Approach 2:
Different levels of processing and analysis are applied to different portions of the data based on their importance and characteristics. Critical security events receive intensive analysis while routine events receive streamlined processing, optimizing the balance between detection accuracy and processing efficiency.
Data Source
AI summary
A method includes generating a polling task that is configured to retrieve raw event data of a subscriber that occurred during a target time span from a third-party security service, computing an optimal polling frequency control value for the polling task in response to generating the polling task, partitioning the target time span into a plurality of distinct sub-intervals of time based on the optimal polling frequency control value computed for the polling task, automatically transmitting, to an application programming interface endpoint of the third-party security service, a plurality of distinct network requests to optimally perform the polling task, receiving the raw event data of the subscriber from the third-party security service that occurred during the target time span in response to transmitting the plurality of distinct network request, and generating one or more security alerts in response to processing the raw event data.