Log Entry Schema Parsing via Common and Type-Specific Metadata
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In computing systems, log files generated by different applications often use varying schemas, making it difficult for log processors to determine the correct schema for extracting specific data, leading to issues in identifying and outputting requested data to users.
Innovation Solution
The system derives specific and common schemas for different log entry types, stores them as metadata, and identifies schema identifiers in log entries to apply the correct schema for parsing and outputting log data, ensuring accurate data extraction and user access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If different schemas are used for different log entry types, then data extraction accuracy is improved, but system complexity increases
Solution Approach 1:
The schema is divided into a common schema (applicable to all log entry types) and type-specific schemas (applicable to individual log entry types). The log processor first applies the common schema to extract shared fields, then applies the appropriate type-specific schema to extract unique fields, thereby improving data extraction accuracy while organizing complexity in a structured manner
Solution Approach 2:
The system derives and stores schemas as metadata in advance before log processing occurs. When a log entry is processed, the log processor retrieves the pre-stored schemas from metadata storage, eliminating the need to analyze and derive schemas during runtime, thus reducing processing complexity while maintaining extraction accuracy
2Reliability
If schemas are stored as metadata, then data integrity is improved, but storage requirements increase
Solution Approach 1:
The common schema serves multiple log entry types simultaneously, defining fields that are shared across all types. This universal schema reduces redundant storage by avoiding duplication of common field definitions for each specific log entry type, thereby maintaining data integrity while minimizing storage requirements
Solution Approach 2:
The schema structure is organized hierarchically with the common schema containing or referencing type-specific schemas. This nested organization allows the system to store schemas efficiently by sharing common definitions while maintaining type-specific variations, reducing overall storage requirements while preserving complete schema information for data integrity
3Productivity
If schema identifiers are identified in log entries, then parsing efficiency is improved, but processing time increases
Solution Approach 1:
The log processor identifies schema identifiers in log entries and retrieves the corresponding schemas from pre-stored metadata before actual data extraction begins. This preliminary retrieval action avoids repeated schema analysis during processing, improving overall parsing efficiency while the upfront metadata storage offsets the additional processing time
Solution Approach 2:
The schema identifier acts as an intermediary between the log entry and the schema definition. Instead of directly analyzing the entire log entry structure to determine the appropriate schema, the system uses the compact identifier to quickly locate and retrieve the full schema from metadata storage, reducing processing time while maintaining parsing efficiency
Data Source
AI summary
A system derives a first schema that is specific to a first log entry type associated with a log code, a second schema that is specific to a second log entry type associated with the log code, and a common schema for the first log entry type and the second log entry type. The system stores the first schema and the common schema in a container for the first log entry type, and the second schema and the common schema in a container for the second log entry type. The system identifies a schema identifier in a log entry corresponding to a system user event. The schema identifier corresponds to a schema in the container for the first log entry type or the container for the second log entry type. The system identifies log data by applying the corresponding schema to the log entry, and outputs the log data.