Cloud Data Intake Orchestration for Scalable Security Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing security intelligence management services face challenges in scalability and efficiency due to varying resource demands from multiple external data sources, leading to over-provisioning and inefficient use of computing resources, as well as frequent redeployment for updates and additions of new data sources, which is error-prone and not highly scalable.
Innovation Solution
A cloud-based data intake and query system uses a combination of a message queuing service and on-demand code execution service to orchestrate the execution of code for accessing and processing data from external sources, allowing independent deployment and scaling of functions based on resource needs, decoupling the execution from the main service.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If computing resources are over-provisioned to handle varying resource demands from multiple external data sources, then service reliability is improved, but resource waste increases
Solution Approach 1:
The system dynamically provisions computing resources based on actual demand from external data sources. The orchestration layer monitors resource usage patterns and automatically scales computing capacity up or down, transitioning from static over-provisioning to dynamic resource allocation that matches actual workload requirements.
Solution Approach 2:
An intermediary orchestration layer is introduced between the security intelligence management service and external data sources. This intermediary manages resource allocation, buffers demand fluctuations, and coordinates computing resource provisioning, preventing both over-provisioning and service disruptions.
2Adaptability or versatility
If the service frequently redeploys to update and add new data sources, then adaptability is improved, but system stability deteriorates
Solution Approach 1:
The system is segmented into independent modular components including the core security intelligence management service, an orchestration layer, and pluggable data source adapters. New data sources can be added by deploying individual adapter modules without requiring redeployment of the entire service, enabling adaptability while maintaining core service stability.
Solution Approach 2:
A universal orchestration framework is implemented that can accommodate multiple types of external data sources through standardized interfaces and pluggable adapters. This multi-functional architecture allows the system to adapt to new data sources without structural changes to the core service, maintaining stability while enhancing versatility.
3Ease of operation
If code execution is tightly coupled with the main service, then ease of operation is improved, but scalability deteriorates
Solution Approach 1:
Code execution functionality is extracted from the main security intelligence management service and placed in a separate on-demand code execution environment. This extraction allows the main service to remain simple and easy to operate while the isolated execution environment provides scalable computing resources for processing external data sources.
Data Source
AI summary
Techniques are described for enabling a cloud-based data intake and query system, and applications designed to interface with a data intake and query system, to use a combination of a message queuing service, an on-demand code execution service, and optionally other services and computing resources provided by a cloud provider network to orchestrate execution of a security intelligence management service in a scalable fashion. The code used to access and process data from individual external services is implemented as independently deployable packages that can be executed by an on-demand code execution service. The execution of such functions can be triggered using a message queueing service, such that the orchestration of functions used to access any number of external services can be managed by a security intelligence management service without the need to provision dedicated computing resources for the entire service.