Cloud Event Log Aggregation for Lower Storage Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The extensive nature of cloud computing event logs poses challenges in efficient storage, management, and analysis due to their large volume, making it difficult to retrieve and process relevant information quickly, especially in identifying significant events amid routine activity.
Innovation Solution
A system and method for generating and storing an aggregated data log by detecting common data fields in multiple log records, merging them to create a merged data record, and storing only the aggregated data log in a repository, thereby reducing data volume and storage costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all cloud computing event logs are stored in detail, then complete monitoring coverage is achieved, but storage resources and data management complexity increase significantly
Solution Approach 1:
The patent extracts and stores only the essential elements of log data (common data fields and unique variations) while discarding redundant information. By identifying common fields across multiple log records and storing them once with references to variations, the system maintains complete monitoring coverage while dramatically reducing stored data volume.
Solution Approach 2:
The system performs preliminary analysis of log records to identify common data fields and patterns before storing the data. By pre-processing logs to detect commonalities and organize them into a condensed format, the system prepares the data in advance for efficient storage and retrieval, avoiding the need to store all raw log details.
2Loss of information
If extensive log data is stored, then comprehensive analysis capability is maintained, but data retrieval and processing speed decrease
Solution Approach 1:
The patent segments log data into common data fields and unique variations, organizing them in a hierarchical structure. This segmentation allows the system to retrieve only the necessary portions of data for specific analysis tasks rather than processing entire log files, significantly reducing retrieval time while maintaining comprehensive analysis capability.
Solution Approach 2:
The system merges multiple log records that share common data fields into a condensed representation, combining redundant information while preserving unique variations. This merging process creates a more compact data structure that enables faster retrieval and processing without losing analytical information.
3Measurement precision
If detailed log records are retained, then troubleshooting accuracy is improved, but storage costs and management overhead increase
Solution Approach 1:
The patent extracts the essential troubleshooting information from detailed log records by identifying common data fields that contain the most valuable diagnostic data. By storing only these essential fields along with references to unique variations, the system maintains troubleshooting accuracy while reducing management overhead.
Solution Approach 2:
The system changes the parameter representation of log data from storing complete detailed records to storing condensed representations with references. This parameter change transforms the data structure from verbose to compact, reducing storage complexity while preserving the ability to retrieve detailed information when needed for troubleshooting.
Data Source
AI summary
A method for generating and storing an aggregated data log is presented. The method includes: accessing a plurality of data log records in a repository; detecting a plurality of records in the repository, wherein each record includes a plurality of data fields; detecting a first data log record of the plurality of data log records having a first data field value in common with a second data log record; detecting in the first data log record a second data field having a second value; detecting in the second data log record the second data field having a third value; generating a merged data record based on: the first data field value, the second value and the third value; generating an aggregated data log based on the merged data record, wherein the aggregated data log includes a plurality of merged data records; and storing the aggregated data log in a repository.