Log Comprehension via Common Schema and ML Knowledge Model
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current log data comprehension techniques fail to effectively map log file data to downstream applications due to inconsistencies in data formats and lack of standardized schema, leading to inefficient data processing and interpretation.
Innovation Solution
The method involves inducting log data into a knowledge base using a log comprehension machine learning knowledge model to create a common log schema, extracting key-value pairs, deriving value features and signals, and applying the model to compare and format data for compatibility with downstream applications, while updating the knowledge base based on feedback.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If log data is processed using conventional techniques without a standardized schema, then data processing can be performed, but data compatibility and interpretation efficiency deteriorate due to format inconsistencies
Solution Approach 1:
The patent implements a universal common log schema that can accommodate multiple log formats and types. The schema defines standardized node types (application, device, event, etc.) and field types (string, number, boolean, etc.) that can represent diverse log data structures, enabling a single processing framework to handle various log sources efficiently
Solution Approach 2:
The patent transforms unstructured or semi-structured log data into structured data by changing the parameter organization from arbitrary formats to standardized schema-defined parameters. This includes mapping log fields to standardized node types and field types, and organizing data hierarchically according to the schema structure
2Measurement precision
If a detailed and comprehensive log schema is created to improve data interpretation, then data comprehension accuracy is improved, but system complexity increases
Solution Approach 1:
The patent segments the log schema into distinct hierarchical levels: log type definitions, node types, field types, and actual log entries. Each level has specific responsibilities and can be independently managed. This segmentation allows comprehensive data comprehension without overwhelming system complexity, as each segment can be processed and validated separately
Solution Approach 2:
The patent introduces an intermediary schema layer that acts as a mediator between raw log data and application-specific interpretations. This schema layer provides standardized node types and field types that translate diverse log formats into a common structure, enabling accurate data comprehension without applications needing to handle raw format variations directly
3Adaptability or versatility
If log data is extracted and formatted in real-time for downstream applications, then data compatibility is improved, but processing time increases
Solution Approach 1:
The patent performs preliminary actions by pre-defining the common log schema structure, node types, and field types before actual log processing occurs. This upfront schema preparation enables faster processing during runtime, as the validation and formatting rules are already established and can be applied systematically without ad-hoc processing decisions
Data Source
AI summary
Techniques are described regarding log data comprehension in a computing environment. An associated computer-implemented method includes inducting data into a knowledge base associated with a log comprehension machine learning knowledge model in order to configure a common log schema. The method further includes extracting at least one key-value pair from a log file including input data associated with at least one downstream application and deriving at least one value feature and any key signal associated with the at least one key-value pair. The method further includes applying the log comprehension machine learning knowledge model in order to compare data associated with the at least one key-value pair extracted from the log file with knowledge base node key-value pair data. The method further includes creating mapping results formatted according to the common log schema and compatible with the at least one downstream application based upon the model application.