Log Message Type Caching for Distributed Analytics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed computing systems face inefficiencies and information loss due to the rapid increase in log/event message volumes, leading to network failures and processing delays, which limit the caching period for message collectors and result in significant data loss.
Innovation Solution
Implementing a method that caches information about log/event message types rather than individual messages, allowing for efficient compression and extended caching periods, thereby tolerating networking failures and delays without significant information loss.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If individual log/event messages are cached, then information retention is improved, but caching period is limited and data loss occurs during network failures
Solution Approach 1:
The patent creates compressed representations (copies) of log message types rather than storing individual messages. Message collectors cache type information that can represent multiple individual messages, reducing storage requirements and enabling extended caching periods while maintaining information integrity during network failures.
Solution Approach 2:
The patent transforms individual log messages into compressed type representations by changing the data structure parameters. Instead of storing complete message content, the system stores condensed type information that can be expanded back into individual messages when needed, allowing much longer caching durations with minimal information loss.
2Duration of action of moving object
If compressed representations of log message types are cached, then caching period is extended, but compression overhead is introduced
Solution Approach 1:
The patent performs compression of log messages into type representations in advance, during the normal message collection process. This preliminary compression action ensures that when network failures occur, the cached compressed representations are already ready for extended storage without requiring real-time compression during failure periods, thus extending effective caching duration while managing processing overhead.
3Quantity of substance
If log message volumes increase, then system monitoring capability is improved, but network failures and processing delays increase
Solution Approach 1:
The patent extracts the essential type information from individual log messages, separating the critical identification data from the complete message content. This extraction allows the system to cache compressed type representations that occupy minimal network bandwidth and storage space, enabling the system to handle increased log volumes while maintaining network reliability and reducing processing delays.
Data Source
AI summary
The current document is directed to methods and systems that efficiently cache log/event messages in log/event-message systems incorporated within distributed computer facilities. The log/event-message systems provide the data-collection, data-storage, and data-querying functionalities that provide a basis for distributed log-analytics systems which, in turn, provide a basis for automated and semi-automated system-administration-and-management systems. Efficient caching significantly extends the period of time that networking failures and log/event-message processing delays can be tolerated by log/event-message systems without significant loss of information. Rather than caching individual log/event messages, message collectors that employ the currently disclosed technologies cache information regarding log/event-message types, leading to efficient compression of the cached information. As a result, message collectors can cache log/event messages without significant loss of information for time periods of several orders of magnitude greater than those achieved by caching individual log/event messages.