Distributed Log Analytics Using Event-Type Condensation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

The management and administration of large, distributed computing systems are increasingly complex, leading to significant inefficiencies and computational overheads due to the large volumes of log/event messages generated, transmitted, and stored, which overwhelm network and data-storage resources.

Innovation Solution

The system efficiently processes and transmits log/event messages by condensing them at the message-collector level, offloading message-processing tasks from message-ingestion-and-processing systems, thereby reducing data volume and network overheads.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of information

If log/event messages are transmitted in full volume from message collectors to message-ingestion-and-processing systems, then complete message data is available for processing, but network overhead and data transmission load increase significantly

Engineering Contradiction:

Improvemessage data completenessVSAvoidnetwork overhead

Core Design Contradiction:

Loss of informationVSLoss of energy

Solution Approach 1:

The patent extracts only the essential event-type information from complete log messages at the message collector level. Instead of transmitting full messages, the system identifies and transmits only the event-type portion that is sufficient for downstream processing, thereby reducing network overhead while maintaining data completeness for processing purposes

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent inverts the traditional approach by having message collectors perform digestion and condensation of messages before transmission, rather than having downstream systems process complete messages. This reversal of processing location reduces the data volume transmitted across the network while ensuring complete event information is captured at the source

Inventive Principle:

Principle #13The other way round (Inversion)

2Productivity

If message collectors transmit all collected messages to message-ingestion-and-processing systems, then all messages are processed centrally, but computational overhead at downstream systems increases

Engineering Contradiction:

Improvemessage processing throughputVSAvoiddownstream system computational load

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent segments the message processing workload by dividing it between message collectors and downstream systems. Message collectors handle the digestion and condensation of messages locally, while downstream systems focus on processing the condensed event-type data. This segmentation reduces the computational complexity at downstream systems while maintaining overall processing throughput

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies preliminary action by having message collectors perform message digestion and condensation before transmission to downstream systems. This pre-processing at the collector level reduces the workload for downstream message-ingestion-and-processing systems, allowing them to focus computational resources on core processing tasks rather than handling raw message volumes

Inventive Principle:

Principle #10Preliminary action

3Reliability

If large volumes of log messages are stored and transmitted, then comprehensive logging coverage is achieved, but data-storage resources are overwhelmed

Engineering Contradiction:

Improvelogging coverageVSAvoiddata volume

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The patent extracts only the essential event-type information from complete log messages for storage and transmission purposes. By identifying and retaining only the critical event-type portion of each message, the system maintains comprehensive logging coverage for reliability while significantly reducing the quantity of data that must be stored and managed

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent changes the parameter of message representation from complete log messages to condensed event-type expressions. This parameter change transforms the data format to retain essential information while reducing volume, allowing comprehensive logging coverage with reduced data-storage requirements

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS12566756B2Efficient event-type-based distributed log-analytics system

Publication Date: 2026.03.03 VMWARE INC

AI summary

The current document is directed to methods and systems that efficiently transmit and process log/event messages within and among distributed computer facilities. By digesting and condensing log/event messages at the message-collector level, the volume of data transmitted from message collectors to message-ingestion-and-processing systems is greatly reduced, which increases system efficiencies by decreasing network overheads and which provides sufficient additional computational bandwidth at the message-collector level to allow message collectors to offload many message-processing tasks from message-ingestion-and-processing system and other downstream processing systems. When the currently disclosed, improved message-collectors carry out message-processing tasks formerly carried out by message-ingestion-and-processing systems and other downstream processing systems, an even greater deduction in the volume of data transmitted from message collectors to message-ingestion-and-processing systems is obtained, further increasing system efficiencies. The decrease in data volume also contributes to increased message-query-processing efficiencies.