Memory Mapped Log File Segments for Asynchronous I/O
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional logging systems are inefficient, causing performance issues due to the expense of logging events, leading to slowed applications and increased latency, especially when disk memory is accessed for writing log files, and require manual configuration, which can hinder troubleshooting and diagnostics.
Innovation Solution
The implementation of a logging system that supports memory mapping and asynchronous file operations, allowing for selective mapping of log file segments, background processing of file operations, and the use of spare files to minimize forced flushes and persist data even during application crashes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional logging systems write log data directly to disk files, then log data is persisted reliably, but application performance deteriorates due to expensive I/O operations and forced flushes
Solution Approach 1:
The log file is divided into multiple segments that can be independently memory-mapped. The logging system writes to specific segments without requiring the entire file to be loaded or flushed to disk, enabling selective I/O operations that improve performance while maintaining data integrity through proper segment management and rotation.
Solution Approach 2:
The patent introduces an intermediary layer between the application and the disk file system. This layer uses memory mapping to create a buffer in memory where log data is first written, then asynchronously flushed to disk. This intermediary buffer decouples the application from direct I/O operations, improving performance while ensuring reliability through controlled flush operations.
2Reliability
If logging systems perform synchronous file operations (opening, closing, deleting), then file operations are completed reliably, but system latency increases and application responsiveness decreases
Solution Approach 1:
The logging system performs preliminary actions by pre-opening log files and maintaining them in memory-mapped state before they are needed for writing. File operations such as opening and closing are performed in advance and cached, so that during normal logging operations, the system does not need to repeatedly open and close files, reducing latency while ensuring file operations are completed reliably.
Solution Approach 2:
The patent maintains continuous logging operations by keeping log files open and memory-mapped throughout the application runtime. Instead of opening and closing files for each logging operation, the system maintains continuous access to the log files in memory, eliminating the latency associated with repeated file open/close operations while ensuring reliable file handling.
3Adaptability or versatility
If logging systems manually configure logging functionality, then logging can be customized for different applications, but system complexity increases and troubleshooting becomes more difficult
Solution Approach 1:
The logging system automatically detects and configures itself based on the application's needs and runtime environment. It performs self-configuration by identifying the application type, determining appropriate log levels and destinations, and setting up memory-mapped log files without requiring manual intervention. This self-service capability maintains adaptability to different applications while reducing configuration complexity and improving troubleshooting by providing consistent default behavior.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods, systems, and computer storage media for providing log files using logging system operations in a logging system. The logging system operations support memory mapping log files and asynchronously managing file operations. The logging system operations support selectively mapping segments of log files during write operations. The logging operations also support performing file operations (e.g., closing, opening, deleting and serializing files) advantageously as background processes. Selective memory mapping specifically includes incrementally mapping new segments of a log file up to a predetermined log file size limit. The logging operations support processing spare files using spare file memory mapping. A spare file replaces an existing log file to continue writing logging data using the spare file. Based on the memory mapping, additional logging system operations of the logging system can be performed including persisting the log file data even when the application crashes, minimizing of forced flushes, and asynchronous file management.