Thread-Local Logging Buffers for Low Interference Diagnostics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The complexity of network computing and storage systems has made diagnosing hardware and software issues challenging, as existing logging implementations can alter system operations and slow down code execution, affecting service availability and performance.
Innovation Solution
Implementing thread-local output buffers with a global declaration list for logging, where log entries are written in a format defined by the log type declarations, and using counter values for timestamping, allowing for efficient logging and real-time monitoring without disrupting system operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Difficulty of detecting and measuring
If logging implementations are added to diagnose hardware and software issues, then diagnostic capability is improved, but system performance and availability deteriorate due to slowed code execution
Solution Approach 1:
The logging system is segmented into separate components: a logging framework that intercepts log calls, thread-local buffers for storing log entries, and a flushing mechanism that transfers logs to permanent storage. This segmentation allows the logging function to be isolated from the main code execution path, minimizing performance impact while maintaining diagnostic capability.
Solution Approach 2:
The system performs preliminary actions by pre-allocating thread-local buffers and establishing logging interceptors before the main code execution begins. Log entries are prepared and stored in memory buffers immediately when generated, and only flushed to permanent storage when necessary, reducing the overhead during critical execution paths.
2Loss of information
If logging implementations are added to capture execution events, then diagnostic information is improved, but system reliability deteriorates due to altered operation
Solution Approach 1:
An intermediary logging framework is introduced between the code execution and the diagnostic storage system. This framework intercepts log calls and manages the flow of information through thread-local buffers, acting as a mediator that captures execution events without directly altering the core code operations, thus maintaining system reliability while preserving diagnostic information.
Solution Approach 2:
The system creates copies of log entries in thread-local buffers before flushing them to permanent storage. This copying mechanism allows the logging system to capture execution events and store them for later analysis without modifying the original code execution or system state, thereby maintaining reliability while preserving complete diagnostic information.
3Difficulty of detecting and measuring
If traditional logging methods are used to monitor system issues, then diagnostic capability is improved, but service availability deteriorates due to performance degradation
Solution Approach 1:
The logging system employs dynamic characteristics by adjusting its behavior based on system conditions. Thread-local buffers are dynamically allocated and managed per thread, and the flushing mechanism dynamically decides when to transfer logs based on buffer fullness and system state. This dynamic approach allows the logging system to adapt to varying workloads and minimize impact on service availability while maintaining issue detection capability.
Data Source
AI summary
Techniques described and suggested herein include systems and methods for logging execution of code using thread-local output buffers. For example, one or more output buffers are allocated to one or more threads executing on a computing system. A global declaration list containing information relating to log types (e.g., verbose log descriptions, templates for specific variables, and the like) may be implemented, and the global declaration list may be generated as part of an initialization process for some or all of the threads. Log events from executing threads may be stored in the output buffers in a fashion conforming to the global declaration list, and may be retrieved asynchronously relative to the executing threads.