Edge Log Collector Master Node Election
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current edge-based log collectors face issues with uninterrupted log flow due to downtime, cumbersome reconfiguration, and inefficiency in processing high volumes of logs, leading to bottlenecks and disruptions in cybersecurity log processing.
Innovation Solution
An edge-based log collecting device with multiple processing nodes that elect a master node using a predetermined algorithm for receiving and sharing logs, allowing for continuous log flow, efficient processing, and seamless reconfiguration without disrupting the log sources, utilizing a floating public IP address for transparent master node switching and load balancing between nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a single log collector is used in the network, then the device complexity is low, but the reliability of log flow is poor because the system stops working when the log collector is down
Solution Approach 1:
The log collector is segmented into multiple independent processing nodes that can operate autonomously. Each node can receive and process logs independently, eliminating the single point of failure. The logs are distributed across multiple nodes, allowing the system to maintain log flow even when individual nodes are down.
Solution Approach 2:
The system changes from a single-collector architecture to a multi-node architecture with dynamic role assignment. Processing nodes can dynamically become master nodes or standby nodes based on their operational status, changing the system's structural parameter to maintain reliability.
2Productivity
If load balancers or routing servers are placed upstream the log collector to handle high volume logs, then the log processing capacity increases, but a bottleneck is created for log flow
Solution Approach 1:
The log processing function is segmented across multiple processing nodes that work in parallel. Each node independently processes a portion of the log flow, eliminating the single-point bottleneck created by traditional load balancers. This distributed architecture maintains high throughput without creating centralization bottlenecks.
3Adaptability or versatility
If the log collector is reconfigured for maintenance or network changes, then the system can be updated, but all log sources must be reconfigured which is cumbersome and time consuming
Solution Approach 1:
The master node function is extracted as a separate, dynamic role from the fixed collector infrastructure. When reconfiguration is needed, only the master node role needs to be transferred to a different processing node, rather than reconfiguring all log sources. This separates the management function from the data collection function.
Solution Approach 2:
The system implements automatic master node election and notification mechanisms. When a new master node is elected, it automatically notifies log sources of its address, eliminating the need for manual reconfiguration. The system provides feedback to maintainers about the current master node status and facilitates automatic updates.
Data Source
AI summary
The invention relates to an edge-based log collecting device for collecting logs from several log sources located in a remote network, called edge-network, and sending the logs to a cloud-based system distant from the edge-network. The device is located in the edge-network, and the device includes several processing nodes for processing logs received from the log sources and sending them to said the system. The device is configured to elect, according to a predetermined election algorithm, one of the processing nodes as a master node configured for receiving the logs from all log sources of the local network, and sharing the logs with the other processing nodes. The invention further relates to a computer program and a device configured to carry out such a method.