Agentless Log Ingestion for Delayed Cybersecurity Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing agent-based log collection solutions introduce operational overhead, consume compute resources, increase the attack surface, and limit retrospective analysis due to real-time log streaming, while log rotation mechanisms can result in permanent data loss, hindering timely and accurate security detection and response in complex environments.
Innovation Solution
A system and method for delayed log ingestion using agentless collection, which involves detecting resources generating local logs, periodically fetching and applying controls to detect cybersecurity issues, and ingesting only new log data, enabling retrospective analysis and reducing data loss through stateful tracking and incremental processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If agent-based log collection is deployed, then log data collection capability is improved, but operational overhead and compute resource consumption increase
Solution Approach 1:
The patent extracts the log collection functionality from agent-based solutions and implements it as an agentless system that directly accesses log files on the file system. This removes the need for deployed agents while maintaining log collection capability, thereby reducing operational overhead and compute resource consumption.
Solution Approach 2:
The patent introduces a file system watcher as an intermediary mechanism that monitors log files for changes and triggers ingestion processes. This mediator enables passive log collection without requiring active agents on target systems, reducing complexity while maintaining reliable data collection.
2Speed
If real-time log streaming is implemented, then forward-looking visibility is improved, but retrospective analysis capability deteriorates
Solution Approach 1:
The patent implements preliminary action by continuously ingesting and storing log data in a searchable repository before security threats are known. This allows historical logs to be retained and analyzed retrospectively when new vulnerabilities are discovered, eliminating the time loss associated with real-time streaming-only approaches.
Solution Approach 2:
The patent ensures continuity of useful action by maintaining both real-time log ingestion for immediate security monitoring and persistent storage for historical analysis. This continuous dual-mode operation preserves both forward-looking visibility and retrospective analysis capabilities simultaneously.
3Quantity of substance
If log rotation mechanisms are configured, then storage management is improved, but data loss risk increases
Solution Approach 1:
The patent implements feedback mechanisms through file system watchers that continuously monitor log files for changes, including rotation events. When logs are rotated or new log files are created, the watcher detects these changes and triggers ingestion processes, ensuring no data is lost during rotation while maintaining efficient storage management through selective ingestion.
Solution Approach 2:
The patent applies dynamics by making the log ingestion process adaptive to log rotation patterns. The system dynamically adjusts its ingestion behavior based on detected log file changes, ensuring that rotated logs are captured while avoiding redundant ingestion of already-processed data, thus balancing storage management with data completeness.
4Reliability
If agents are deployed on virtual machines, then log collection coverage is improved, but attack surface increases
Solution Approach 1:
The patent extracts log collection functionality from deployed agents and implements it through agentless file system monitoring. This removes the attack surface associated with installed agents while maintaining comprehensive log collection coverage by directly accessing log files on the file system through existing permissions.
Data Source
AI summary
A system and method for ingesting delayed logs for cybersecurity detection is presented. The method includes detecting a resource deployed in a computing environment, the resource configured to generate a local log on a disk of the resource; periodically fetching the local log from the disk, the local log including a plurality of event records written by at least a software application executed on the resource; applying a control on the fetched local log; and detecting a cybersecurity issue on the resource based on a result of applying the control.


