Data Collection Connector Abstraction for Secure Scalable Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern data centers face challenges in efficiently and securely collecting and analyzing vast amounts of machine-generated data from diverse sources, as existing systems often require pre-processing and have limitations in scalability and access control, particularly when third-party connectors are involved.
Innovation Solution
The solution involves abstracting connector functionality from data-collecting processes, enabling separate execution and utilizing a discover and collect process to facilitate scalable and secure data collection, allowing third-party connectors to operate securely and efficiently, and enabling parallel execution to enhance data collection efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If connector functionality is integrated into data-collecting processes, then data collection can be performed, but security is compromised due to unauthorized access to restricted resources
Solution Approach 1:
The system divides connector functionality into a separate discover process that runs independently from the main data-collecting process. This segmentation allows the connector to discover and collect data while the main process maintains security controls, resolving the contradiction between enabling third-party connectors and maintaining security.
Solution Approach 2:
The connector functionality is extracted from the main data-collecting process and placed in a separate discover process. This extraction prevents the connector from accessing restricted resources in the main process while still allowing it to perform its data discovery and collection functions independently.
2Productivity
If data collection is performed in a single process, then the system is simpler, but efficiency is reduced due to lack of parallel execution
Solution Approach 1:
The data collection system is segmented into multiple independent processes: a discover process that discovers data to collect and a collect process that collects the data. These processes can execute in parallel, improving efficiency while maintaining manageable complexity through clear process separation.
Solution Approach 2:
The discover process performs preliminary actions by discovering what data needs to be collected before the collect process executes. This preliminary discovery phase allows the system to prepare data collection parameters and structures in advance, enabling more efficient parallel execution of the actual data collection.
Data Source
AI summary
This technology is directed to facilitating scalable and secure data collection. In particular, scalability of data collection is enabled in a secure manner by, among other things, abstracting a connector(s) to a pod(s) and/or container(s) that executes separate from other data-collecting functionality. For example, an execution manager can initiate deployment of a collect coordinator on a first pod associated with a first job and deployment of a first connector on a second pod associated with a second job separate from the first job of a container-managed platform. The collect coordinator can provide a data collection task to the first connector deployed on the second pod of the second job. The first connector can then obtain the set of data from the data source and provide the set of data to the collect coordinator for providing the set of data to a remote source.