Network Event Capture System for Storage Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional network monitoring systems face significant challenges due to the voluminous nature of network event notifications, which leads to high storage costs and decreased data accessibility, as they often summarize or abridge data to mitigate these issues, potentially losing important diagnostic information.
Innovation Solution
A system that captures and stores network event notifications in their entirety using a hierarchical file system, with indices to quickly locate specific data elements, allowing for the creation of summaries and aggregate summaries without the need for full data access, thereby preserving raw data for improved diagnostics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If network event notifications are stored in their entirety in a conventional database, then complete diagnostic information is preserved, but storage costs increase significantly and data accessibility decreases
Solution Approach 1:
The patent segments network event data into two categories: summarized/normalized data for routine monitoring and storage-efficient access, and complete raw notifications preserved for detailed forensic analysis. This segmentation allows the system to maintain both data completeness and storage efficiency by storing only essential metadata in the database while preserving full notifications separately.
Solution Approach 2:
The patent extracts and stores only the most critical elements of network notifications (such as event type, timestamp, and key parameters) in the conventional database, while removing redundant or less critical information. This extraction approach maintains data accessibility and reduces storage requirements while preserving sufficient information for most diagnostic purposes.
2Quantity of substance
If network event notifications are summarized or abridged to reduce storage requirements, then storage costs decrease, but important diagnostic information may be lost
Solution Approach 1:
The patent applies different quality levels of data storage to different usage scenarios: summarized data with lower information content is used for routine monitoring and quick access, while complete raw notifications with full information quality are preserved for detailed forensic analysis. This local quality approach ensures that each data representation is optimized for its specific purpose.
Solution Approach 2:
The patent performs preliminary summarization and normalization of network events before storing them in the database, extracting only the essential metadata needed for routine monitoring. The complete raw notifications are preserved separately in advance, ensuring that full diagnostic information is available when needed without requiring storage of all detailed data in the primary database.
3Loss of information
If complete network event data is stored for forensic analysis, then diagnostic capability is improved, but data accessibility and retrieval time decrease
Solution Approach 1:
The patent implements a dynamic data access strategy where the system automatically selects between querying summarized data for routine monitoring (faster access) and retrieving complete raw notifications for detailed forensic analysis (slower but more comprehensive access). This dynamic approach optimizes retrieval time based on the specific diagnostic needs of each query.
Solution Approach 2:
The patent introduces an intermediary layer between the database and the user interface that provides summarized and normalized views of network events for quick access and routine monitoring. This intermediary layer acts as a mediator that translates between the compact stored data and the detailed information needed for analysis, enabling fast retrieval for common operations while maintaining access to complete data when required.
Data Source
AI summary
Methods and apparatus are provided to monitor and analyze activity occurring on a networked computer system. In some embodiments, a method is provided for capturing, in a data structure, at least a portion of a notification describing a network event provided by a node on a computer network, identifying a data element (e.g., an IP address of the node) within the notification, and updating an index and/or summary based on the data element. The data structure may be stored in a file system maintained on a site, and sites may exchange information related to the notification data stored on each. In some embodiments, a query which is issued to a site may be processed using data transferred from other sites, and/or may be split into one or more additional queries which may be transmitted for processing to other sites.