Disk Logging Method Using Dynamic Batch Sizing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing disk logging methods face challenges in achieving high throughput and low latency for small data payloads due to mismatches between fine-grained logging and modern file systems, high per-operation overheads, and inefficient use of disk resources, particularly with hard disk drives being considered unsuitable for logging due to slow performance and high costs of alternative solutions like SSDs.
Innovation Solution
A disk logging method that dynamically adjusts batch sizes and uses a traditional hard disk drive to aggregate logging operation requests, optimizing the disk write pipeline by continuously submitting write requests with consecutive target addresses, thereby keeping the disk fully occupied and minimizing latency and maximizing throughput, without requiring detailed knowledge of disk geometry.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a flash based memory such as NVRAM is used for logging disk, then reliability is improved, but cost increases
Solution Approach 1:
The patent uses HDDs as logging disks, which are cheaper than SSDs/NVRAM, accepting that they are not permanent but sufficient for the temporary logging function. The logging disk is intentionally designed as a disposable component that can be easily replaced or reformatted, eliminating the need for expensive durable storage solutions.
2Ease of manufacture
If an HDD is used for logging disk, then cost is reduced, but speed deteriorates
Solution Approach 1:
The system pre-allocates and pre-writes to the logging disk in a controlled manner, preparing the disk surface in advance to minimize seek times during actual logging operations. The disk is formatted and initialized beforehand with optimal parameters for logging workloads.
Solution Approach 2:
The patent changes the operational parameters of the HDD by adjusting batch sizes, write ordering, and disk queue management to optimize for logging workloads rather than general-purpose storage. The system modifies how the disk is accessed and controlled to extract maximum performance from the mechanical drive.
3Productivity
If fine-grained logging operations are performed, then productivity is improved, but device complexity increases
Solution Approach 1:
The patent extracts the logging function from the complex file system layer and implements it as a dedicated subsystem with its own simplified queue management and write handling. The logging disk operates independently from the main file system, eliminating the need for complex file system operations and metadata management.
Solution Approach 2:
The system segments the logging operations into separate queues and processing stages, isolating the logging function from general file system operations. This segmentation allows fine-grained logging to be handled efficiently without the overhead of conventional file system processing.
4Adaptability or versatility
If multiple processing steps are added to the data path, then adaptability is improved, but loss of time increases
Solution Approach 1:
The patent implements continuous writing to the logging disk without interruption, maintaining a steady stream of data writes that keeps the disk busy and eliminates idle time. The system continuously submits write requests to maintain optimal disk utilization and minimize latency.
Solution Approach 2:
The system uses periodic flushing and batched writing operations to manage the data path efficiently, organizing writes into regular cycles that minimize processing steps while maintaining flexibility. The periodic nature of the operations allows for optimized timing and reduced overhead.
Data Source
AI summary
The present disclosure proposes a disk logging method configured for an electronic device comprising a temporary non-volatile storage medium to log data from a volatile memory to said first storage medium, and the method includes the elements of aggregating data from applications of the electronic device in a queue, transferring the aggregated data to a per device queue targeted toward a native queue of the storage medium, writing the data stored in the native queue of the storage medium into a disk platter of the storage medium, and transmitting an interrupt in response to the completion of the writing of the data to the disk platter, wherein the first batch size is dynamically adjusted such that the step of writing the data to the platter takes more time than the step of transferring the data from the per device queue to the native queue of the storage medium.