Dynamic Log Analytics Engine for Real-Time Threat Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional security monitoring and analysis approaches in hyperscaling virtualized remote environments are unable to process vast volumes of data in real-time, leading to delayed detection and disruption of attacks, which allows attackers to cause exponential damage.
Innovation Solution
A cloud-based analytics engine system with subsystems for data ingestion, filtering, real-time parsing, data enrichment, aggregation, and correlation triggers, capable of dynamically processing and responding to security threats by extracting relevant data elements and generating actionable insights.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional security monitoring approaches ingest and analyze all log data streams, then comprehensive security coverage is achieved, but processing speed and response time deteriorate due to analytic latency
Solution Approach 1:
The patent extracts and prioritizes only the most critical log events for immediate analysis while routing less critical events to batch processing. The system identifies high-value security events (authentication failures, privilege escalations, data access patterns) and processes them in real-time, while extracting and storing less urgent events for later analysis, thereby reducing analytic latency for critical threats
Solution Approach 2:
The patent segments the log data processing pipeline into multiple specialized components: real-time stream processing for critical events, batch processing for comprehensive analysis, and hierarchical alerting tiers. This segmentation allows different processing speeds and depths for different event types, maintaining reliability for critical events while improving overall system throughput and reducing latency
2Device complexity
If static parsing methods are used to process log data streams, then system complexity is reduced, but the ability to adapt to changing attack patterns deteriorates
Solution Approach 1:
The patent implements dynamic parsing rules that automatically adjust based on detected attack patterns and threat intelligence. The system learns from new attack vectors and modifies its parsing and analysis behavior in real-time, enabling adaptation to evolving threats while maintaining manageable system complexity through automated rule generation and prioritization
3Reliability
If massive amounts of log data are stored and indexed for analysis, then detection accuracy is improved, but resource consumption and processing overhead increase
Solution Approach 1:
The patent applies local quality by storing and indexing only the specific log fields and event types most relevant to security threat detection, rather than maintaining complete copies of all log data. The system identifies and prioritizes critical attributes (user identifiers, timestamp ranges, event categories, source/destination addresses) for efficient indexing, while storing less critical data in compressed or aggregated forms, thereby improving detection accuracy for relevant threats while reducing overall resource consumption
Data Source
AI summary
A configurable system and method for automatically taking in streams of log data from various sources, dynamically parsing, normalizing the data and routing it to subsystems of an analytics engine. The routed data may undergo aggregating and other enrichment based on content, rules and data, so as to generate useful event observations, which may recursively be fed back into the system's data ingestion stream to further enhance the usefulness of the system's outputs, in real-time, in the context of computer system and data security.