Structured Logging System with Event Filtering and Routing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Managing and routing diverse streams of program-related events in a computer program is challenging due to the large number and variety of events, making it difficult for developers, testers, and services to access valuable data for testing, debugging, and analytics.
Innovation Solution
A structured logging system that uses log handlers with filtering mechanisms to select and route relevant event log entries to appropriate event sinks, allowing for customized data handling and flexible event management, including runtime configuration of log handlers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If all program events are collected and reported to event sinks, then complete event data is available for analysis, but system performance degrades due to redundant processing and network overhead
Solution Approach 1:
The filtering mechanism extracts only the necessary event log entries from the complete event stream based on specified criteria, sending only relevant events to event sinks. This extraction process maintains data completeness for needed events while eliminating redundant processing of unnecessary events, thus resolving the contradiction between complete data availability and system performance.
Solution Approach 2:
Instead of processing all events equally, the system applies partial action by selectively processing only those events that match the filtering criteria. This partial processing approach maintains sufficient event data for analysis while reducing overall processing load and network overhead, thereby improving application responsiveness without sacrificing necessary data completeness.
2Productivity
If a filtering mechanism is implemented to select specific event log entries, then redundant processing is reduced and performance improves, but system complexity increases
Solution Approach 1:
The logging system is segmented into distinct functional components: event sources, log handlers with filtering mechanisms, and event sinks. This segmentation allows the filtering functionality to be added as a modular component rather than integrating complexity throughout the entire system, thus improving processing efficiency while managing system complexity through clear separation of concerns.
Solution Approach 2:
The log handler with filtering mechanism serves as an intermediary component between event sources and event sinks. This intermediary layer handles the complexity of event selection and filtering, allowing event sources to continue generating events and event sinks to consume filtered results without either side needing to understand the filtering logic, thus managing complexity locally while improving overall efficiency.
3Adaptability or versatility
If multiple log handlers with different filtering criteria are deployed, then specific event types can be routed to appropriate consumers, but device complexity increases
Solution Approach 1:
The logging system is designed to be dynamic, allowing log handlers with different filtering criteria to be added, removed, or configured at runtime based on specific needs. This dynamic configuration capability enables flexible event routing for different event types without requiring a fixed complex architecture, as the system can adapt its handler configuration to match actual requirements.
Solution Approach 2:
The log handler component is designed as a universal multi-functional unit that can handle different filtering criteria and route to different event sinks. This universal handler design means that the same basic component structure can serve multiple purposes with different configurations, reducing overall system complexity compared to having specialized handlers for each event type.
Data Source
AI summary
The described technology is directed towards a structured logging technology in which events corresponding to program execution are received in a structured format and logged based upon filtering of those events. A log handler is associated with a filtering mechanism that determines whether each event matches filtering criteria and is thus to be logged by the log handler. The log handler provides matching logged events to an event sink, such as an analytic tool that consumes the events for analysis.