Raw Log Server Data Segmentation and Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems for handling log data from network devices face challenges in storing and retrieving raw log data due to its high volume and the reliability issues caused by delayed storage, leading to logistical problems and reduced data integrity.
Innovation Solution
A system where raw log data is received by a raw log server, stored in a database, and sent to a networked log data analyzer for parsing, summarizing, and reporting, with processes for archiving and purging, and integrating local and remote data using different protocols, ensuring data integrity through encryption and compression over a wide area network.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If raw log data is stored in complete form in a database, then data integrity is maintained, but storage capacity and retrieval efficiency are reduced
Solution Approach 1:
The patent segments log data into two categories: raw log data stored in a first database and parsed/summarized log data stored in a second database. This segmentation allows the raw data to be preserved for integrity while the parsed data provides efficient retrieval capabilities, resolving the contradiction between maintaining complete data and enabling efficient access.
2Loss of energy
If raw log data is delayed for storage, then network bandwidth is conserved, but data reliability is reduced
Solution Approach 1:
The patent introduces a log data analyzer as an intermediary component that receives raw log data, parses it, and stores both the raw and parsed versions in separate databases. This intermediary processing allows the system to maintain data reliability by immediately storing raw data while also enabling efficient retrieval through parsed summaries, without requiring delayed storage that would compromise integrity.
3Productivity
If log data is parsed and summarized before storage, then retrieval efficiency is improved, but data completeness is lost
Solution Approach 1:
The patent merges two storage approaches by maintaining both raw log data and parsed/summarized log data in separate but complementary databases. The raw data database preserves complete information for audit and forensic purposes, while the parsed data database provides efficient retrieval for monitoring and analysis, thus combining the benefits of both approaches without losing data completeness.
Data Source
AI summary
A system and method is disclosed for collecting, storing and reporting raw log data from log-producing devices such as firewalls and routers. The log-producing devices may be both local and remote—i.e., linked to a raw log server via a LAN and/or a WAN. A log data analyzer at a remote location gathers log data from devices at that remote location into time-defined sets and then sends those sets over a WAN (which may be the Internet) to a raw log server using a first protocol. Local log-producing devices may send their log data to the log data analyzer via a LAN using a second protocol. The log data analyzer forwards the raw log data local devices to an appropriate log data analyzer for parsing, summarizing and storage in one or more databases. The raw log server combines local and remote sets of raw log data for a given time period and stores them in a storage area of raw log data. A central management station is used to query the various databases in the system and to merge database reports into a single report for display.