Ephemeral High-Cardinality Log Processing Before Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Processing and storing large quantities of high cardinality log data increases storage costs and computing resource consumption, leading to adverse impacts on computing systems.
Innovation Solution
An aggregation layer is implemented at the network level to deduplicate high cardinality log data before storage, reducing throughput and storage costs by aggregating and filtering unique log data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If log data is processed and stored in large quantities, then monitoring and analysis capability is improved, but storage costs and computing resource consumption increase
Solution Approach 1:
The patent extracts only the unique and essential log data from the high cardinality data stream using deduplication algorithms. By taking out only the necessary information (unique logs) and discarding duplicates, the system maintains monitoring capability while significantly reducing storage requirements and computing resource consumption.
Solution Approach 2:
The system changes the parameters of log data processing by applying deduplication algorithms that compare log entries based on specific parameters (timestamp, source, message content). By changing how logs are processed from storing everything to storing only unique entries, the system resolves the contradiction between comprehensive monitoring and resource efficiency.
2Loss of information
If high cardinality log data is processed, then data completeness is improved, but system performance deteriorates
Solution Approach 1:
The patent extracts unique log characteristics and metadata (timestamp, source, log level, message) while filtering out duplicative content. This extraction approach preserves the essential information needed for analysis while removing redundant data that degrades system performance, thereby maintaining data completeness without sacrificing productivity.
Solution Approach 2:
The system performs preliminary deduplication processing before data is fully ingested into the storage system. By applying deduplication algorithms in advance during the log ingestion phase, the system reduces the total volume of data that needs to be processed downstream, improving overall system performance while maintaining complete logging capability.
3Reliability
If all log data is stored, then audit trail completeness is improved, but storage costs increase
Solution Approach 1:
The patent extracts and stores only unique log entries that contribute to audit trail completeness. By identifying and retaining only the first occurrence of each unique log pattern while discarding subsequent duplicates, the system maintains a complete audit trail of unique events without incurring the storage costs of storing every single log instance.
Solution Approach 2:
The system changes the storage approach by storing logs as structured data with key parameters (timestamp, source, log level, message template) rather than storing complete raw log strings. This parameter-based storage method reduces storage requirements while maintaining the ability to reconstruct complete audit trails when needed.
Data Source
AI summary
Systems and methods for ephemeral processing of high cardinality data. The system can receive log data indicative of metrics associated with a computing system, wherein the log data is received by an aggregation layer. The method includes aggregating the log data by deduplicating the plurality of logs using one or more aggregation parameters, wherein the one or more aggregation parameters are configurable to increase or decrease a level of deduplication. The method includes, in response to aggregating the log data, determining deduplicated log data including one or more unique logs indicative of unique metrics associated with the computing system. The method includes transmitting the deduplicated log data to a storage system.