Buffered Firewall Log Processing for Analytics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Firewall systems face significant costs in data storage and processing due to the large volume of logs generated for threat analysis, requiring substantial computational resources and storage capacity.
Innovation Solution
A system utilizing a map-reduce algorithm to create buffered batches of firewall log data, processed in fixed intervals, which reduces data complexity and cost by using cloud technologies like AWS Glue and Elasticsearch for data analytics, allowing for efficient data aggregation, storage, and reporting without modifying the existing firewall system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If firewall systems store and process large volumes of log data for threat analysis, then threat detection capability is improved, but data storage and processing costs increase significantly
Solution Approach 1:
The patent segments firewall log data into buffered batches processed in fixed time intervals (e.g., 15-minute periods). This segmentation allows the system to process data in manageable chunks rather than handling the entire log volume at once, reducing processing costs while maintaining comprehensive threat detection capability through continuous interval-based analysis.
Solution Approach 2:
The system performs preliminary data aggregation and processing in advance by creating buffered batches before threat analysis is needed. This preliminary action consolidates data from multiple time intervals into summarized datasets, reducing the complexity and cost of subsequent threat detection operations while preserving the necessary analytical information.
2Speed
If firewall systems process large volumes of log data in real-time, then threat response speed is improved, but computational resource requirements increase
Solution Approach 1:
The patent implements periodic processing of firewall logs in fixed time intervals (e.g., every 15 minutes). This periodic action allows the system to maintain timely threat detection and response while distributing computational workload over time rather than processing all data simultaneously, thereby reducing peak computational resource requirements.
Solution Approach 2:
The system performs preliminary data aggregation and buffering in advance, preparing processed data batches before threat analysis is required. This preliminary processing reduces the computational burden during actual threat detection operations, allowing for faster response times with lower instantaneous computational resource consumption.
3Loss of information
If firewall systems store comprehensive log data for analysis, then data availability for reporting is improved, but storage infrastructure complexity increases
Solution Approach 1:
The patent segments stored data into organized buffered batches corresponding to specific time intervals. This segmentation creates a structured storage approach where data is arranged in manageable, time-based units, simplifying the storage infrastructure while ensuring comprehensive data availability for reporting across different time periods.
Solution Approach 2:
The system performs preliminary data aggregation and organization in advance, creating pre-processed data batches that are ready for reporting. This preliminary action reduces the complexity of storage infrastructure by pre-organizing data in a report-ready format, while maintaining complete data availability through the buffered batch structure.
Data Source
AI summary
A system for firewall data log processing, comprising a firewall logging system operating on a first processor and configured to cause the first processor to receive firewall log data and to process the firewall log data on a periodic basis to reduce the size of the firewall log data and a firewall reporting system operating on a second processor and configured to process the reduced size firewall log data to generate a report on a user interface that includes one or more analytics from the reduced size firewall data.