Log File Query Processing Using Event-Type Metadata

Solution Overview

Problem

In large distributed computing systems, efficiently processing queries against log-file entries is challenging due to the computational expense of opening and reading numerous containers, many of which do not contain relevant event types, leading to significant overhead and inefficiency.

Innovation Solution

The use of event-type metadata associated with log-file-entry containers to avoid opening and reading containers that do not contain relevant event types, employing a hashmap data structure to identify event types without accessing the container contents, and a hierarchical container-directory structure to facilitate query processing.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If all log-file-entry containers are opened and read to execute a query, then complete query results are obtained, but computational and temporal costs increase significantly

Engineering Contradiction:

Improvequery result completenessVSAvoidquery processing time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system performs preliminary actions by maintaining metadata that pre-describes the event types contained in each log-file-entry container. This metadata is created and updated in advance, allowing the query-processing subsystem to determine container relevance before opening the container, thus avoiding unnecessary I/O operations and reducing query processing time while ensuring complete results are obtained from relevant containers only

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system introduces metadata as an intermediary layer between the query and the log-file-entry containers. This metadata acts as a filter or mediator that determines which containers should be accessed based on event-type matching, preventing direct access to irrelevant containers and significantly reducing the computational overhead of query execution

Inventive Principle:

Principle #24Intermediary (Mediator)

2Measurement precision

If all log-file-entry containers are opened and read to execute a query, then complete query results are obtained, but system resource burden increases

Engineering Contradiction:

Improvequery result completenessVSAvoidsystem resource consumption

Core Design Contradiction:

Measurement precisionVSUse of energy by moving object

Solution Approach 1:

The system performs preliminary actions by maintaining metadata that pre-describes the event types contained in each log-file-entry container. This metadata is created and updated in advance, allowing the query-processing subsystem to determine container relevance before opening the container, thus avoiding unnecessary I/O operations and reducing query processing time while ensuring complete results are obtained from relevant containers only

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system introduces metadata as an intermediary layer between the query and the log-file-entry containers. This metadata acts as a filter or mediator that determines which containers should be accessed based on event-type matching, preventing direct access to irrelevant containers and significantly reducing the computational overhead of query execution

Inventive Principle:

Principle #24Intermediary (Mediator)

3Productivity

If metadata is maintained for each log-file-entry container, then query processing efficiency is improved, but device complexity increases

Engineering Contradiction:

Improvequery processing efficiencyVSAvoiddata structure complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The system applies local quality by maintaining metadata at the container level rather than processing individual log entries. Each container has its own metadata describing its event types, allowing the system to make quick decisions about container relevance without examining individual entries. This localized approach simplifies the overall system architecture while improving query processing efficiency

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The system performs preliminary actions by maintaining metadata that pre-describes the event types contained in each log-file-entry container. This metadata is created and updated in advance, allowing the query-processing subsystem to determine container relevance before opening the container, thus avoiding unnecessary I/O operations and reducing query processing time while ensuring complete results are obtained from relevant containers only

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS10776439B2Efficient log-file-based query processing

Publication Date: 2020.09.15 VMWARE INC

AI summary

The current document is directed to systems, and methods incorporated within the systems, that execute queries against log-file entries. A monitoring subsystem within a distributed computer system uses query results during analysis of log-file entries in order to detect changes in the state of the distributed computer system, identify problems or potential problems, and predict and forecast system characteristics. Because of the large numbers of log-file-entry containers that may need to be opened and processed in order to execute a single query, and because opening and reading through the entries in a log-file-entry container is a computationally expensive and time-consuming operation, the currently disclosed systems employ event-type metadata associated with log-file-entry containers to avoid opening and reading through the log-file entries of log-file-entry containers that do not contain log-file entries with event types relevant to the query.