Asynchronous Actor-Based Log Data Parser
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Manufacturers face difficulties in processing, storing, and utilizing log data from complex software and hardware systems, which often contain unstructured or semi-structured information, to detect bugs, analyze performance, and predict failures in a timely and efficient manner.
Innovation Solution
A framework that compiles domain-specific language-based descriptions of log formats into parsers, using an actor structure and message bus infrastructure to process log data asynchronously and in parallel, allowing for high-volume log data processing with minimal disruption, even if individual actors encounter errors, while other actors continue processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If log data is processed using traditional sequential methods, then processing simplicity is maintained, but processing speed and efficiency deteriorate when handling high volumes of log data
Solution Approach 1:
The patent divides the log data processing system into multiple independent actor components, each capable of processing different portions of log data simultaneously. These actors are organized in a hierarchical structure with root actors, intermediate actors, and leaf actors, allowing parallel processing while maintaining modular complexity management.
Solution Approach 2:
The patent transitions from sequential single-threaded processing to multi-dimensional parallel processing by introducing multiple actor instances that can execute concurrently. This dimensional shift from time-sequential to space-parallel processing enables handling high volumes of log data simultaneously without proportionally increasing system complexity.
2Productivity
If parallel processing is used to increase processing throughput, then productivity improves, but system reliability deteriorates due to potential errors in individual processing units
Solution Approach 1:
The patent implements fault tolerance mechanisms where parent actors monitor and manage child actors proactively. When errors occur in individual actor instances, the system has pre-established error handling protocols that allow other actors to continue processing without disruption, cushioning the impact of failures on overall system reliability.
Solution Approach 2:
The patent applies different quality characteristics to different parts of the processing system. Individual actor instances can be independently configured with specific error handling strategies, while the overall system maintains high reliability through the collective redundancy and independence of multiple actors. Each actor's local failure does not compromise the global system quality.
3Adaptability or versatility
If log data from multiple sources and formats is processed simultaneously, then adaptability improves, but processing complexity increases
Solution Approach 1:
The patent creates a universal actor framework that can handle multiple log formats and sources through a common interface. Different parser types can be instantiated as actor children, all managed by parent actors using the same coordination mechanisms. This multi-functional design allows the system to adapt to diverse log formats without increasing the fundamental processing complexity.
Solution Approach 2:
The patent segments the handling of different log formats into independent parser actors, each specialized for a specific format. This segmentation allows each parser to be optimized independently while the overall system manages diversity through the modular actor architecture, preventing format-specific complexity from propagating system-wide.
4Speed
If synchronous processing is used to ensure data consistency, then data accuracy is maintained, but processing speed deteriorates due to waiting for completion of each processing step
Solution Approach 1:
The patent introduces message queues and event streams as intermediary components between actors. These intermediaries buffer and manage data flow, allowing actors to process data asynchronously while maintaining consistency through the intermediary's coordination. Data consistency is preserved through the structured message passing mechanism rather than synchronous blocking.
Solution Approach 2:
The patent enables continuous processing by allowing actors to operate asynchronously without blocking on completion of other actors. The system maintains data consistency through the actor model's inherent message passing semantics while keeping all processing actions continuous and non-blocking, achieving both speed and consistency.
Data Source
AI summary
Techniques to process machine generated log data are disclosed. In various embodiments, a parser definition associated with a set of log data is determined. The parser definition is compiled to create an instance of a parser to parse the set of log data. The parser has a hierarchical structure comprising a plurality of hierarchically related nodes, each of at least a subset of said nodes having associated therewith one or more actors each configured to parse data associated with that node. At least a portion of the set of log data is sent to the parser instance prior to compilation of said parser instance being completed. A first node of the parser instance is configured to receive and parse log data associated with the first node even if compilation of the parser definition has not been completed with respect to a second node of said parser instance.