Concurrent File Logging With Lock-Free Log Rolling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Concurrent write operations by multiple threads to a common data store often result in data loss, corruption, and delays due to conflicts and inefficiencies in existing systems, such as separate file allocation or thread locking, which degrade system performance.
Innovation Solution
A method and system allowing concurrent write operations by multiple threads to a common data store by determining which thread is responsible for ensuring data is written and designating a new store when needed, minimizing reliance on CAS-type operations and file locks, and ensuring minimal delay in file rolling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If separate files are allocated for each thread to avoid conflicts, then data loss and corruption are prevented, but memory and storage requirements increase
Solution Approach 1:
The patent divides the file writing process into segments by allowing each thread to write to specific regions of a shared log file using offset pointers. Thread A writes to offset 0-100, Thread B writes to offset 100-200, and so on. This segmentation approach prevents conflicts while maintaining a single consolidated log file, thus preserving data integrity without increasing storage requirements.
Solution Approach 2:
The patent introduces a new dimension to file writing by using offset pointers and file positions rather than allocating separate files for each thread. Threads write to different positions within the same file simultaneously, transforming the problem from a one-dimensional (separate files) to a multi-dimensional (position-based) solution space, achieving both data integrity and storage efficiency.
2Reliability
If thread locking is used to enable only one thread to write at a time, then data corruption is avoided, but productivity and efficiency decrease
Solution Approach 1:
The patent enables continuous writing operations by multiple threads simultaneously to a shared log file without locking. Each thread continuously writes to its assigned offset range, eliminating the stop-start nature of locked systems. This maintains data consistency through coordinated offset management while preserving continuous productive action from all threads.
Solution Approach 2:
The patent introduces offset pointers and file position tracking as intermediaries between threads and the log file. These intermediaries coordinate write operations without requiring thread locking, allowing simultaneous access while maintaining data consistency. The intermediary mechanism manages the shared resource without blocking threads, thus preserving productivity.
3Ease of operation
If CAS-type operations are used for synchronization, then concurrent access is managed, but software loops and delays increase
Solution Approach 1:
The patent performs preliminary allocation of offset ranges to threads before writing begins. Each thread is assigned specific offset ranges in advance, eliminating the need for runtime synchronization and CAS operations. This preliminary action prevents conflicts before they occur, removing software loops and delays entirely while maintaining ease of concurrent access management.
Data Source
AI summary
A data storage system for use with a multi-threaded processing system receives concurrent requests to store data to a common data store, and efficiently and securely swaps an active data store for a new data store while avoiding conflicts arising from multiple threads attempting to swap a same data store and minimizing reliance on operations that re-attempt actions upon failure of an attempted action, thereby improving performance of the data storage system and also the multi-threaded processing system.