Parallel Batch Processing of Write Transactions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional file systems face performance overhead due to batch processing of write requests, which leads to bursts of writes to storage devices, reducing overall processing efficiency and causing delays in data commitment.
Innovation Solution
Implementing a system and method for parallel processing of write requests by grouping them into separate transactions or 'trains' with 'cars' of transaction records, allowing simultaneous writing to storage areas, thereby avoiding bursts and maintaining consistent performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If batch processing is used to commit multiple I/O requests collectively, then data consistency and reliability are improved, but storage area receives write bursts that increase overhead and reduce processing performance
Solution Approach 1:
The patent segments the batch processing into multiple independent thread pools, where each thread pool processes a subset of I/O requests independently. This segmentation allows parallel processing of write operations, preventing write bursts from overwhelming the storage area while maintaining data consistency through coordinated commit operations across multiple threads.
Solution Approach 2:
The patent implements dynamic thread pool management where the number of active threads and their processing capabilities can be adjusted based on system load and storage area capacity. This dynamic adaptation allows the system to optimize between data consistency requirements and processing performance by adjusting the degree of parallelism in real-time.
2Device complexity
If sequential batch processing is used, then system complexity is reduced, but processing time increases and throughput decreases
Solution Approach 1:
The patent divides the processing system into multiple independent thread pools, each capable of processing I/O requests concurrently. This segmentation enables parallel execution of batch processing operations, significantly reducing processing time and increasing throughput while maintaining manageable system complexity through standardized thread pool architecture.
Solution Approach 2:
The patent transitions from sequential single-threaded processing to multi-threaded parallel processing, adding the dimension of concurrency to the batch processing system. This dimensional change allows multiple batches to be processed simultaneously, reducing overall processing time without proportionally increasing system complexity.
3Productivity
If write requests are processed concurrently in parallel, then processing performance and throughput are improved, but data consistency and transaction integrity may be compromised
Solution Approach 1:
The patent segments write requests into distinct batches assigned to different thread pools, with each thread pool responsible for a specific subset of transactions. This segmentation enables concurrent processing of multiple batches while maintaining data consistency through proper batch isolation and coordinated commit mechanisms that prevent interference between parallel operations.
Solution Approach 2:
The patent implements feedback mechanisms where thread pools monitor and report on their processing status, commit outcomes, and any consistency issues. This feedback enables the system to detect and resolve data consistency problems in parallel processing scenarios, ensuring transaction integrity while maintaining high throughput.
Data Source
AI summary
A system and method is provided for parallel processing of multiple write requests to write data associated the multiple write requests to a storage area concurrently. The file system receives a series of write request from one more applications executing on the computing device. The file system includes one or more processing modules that selectively groups writes request into lists or trains. After the processing modules begin processing first threads associated with writes in a first train, the processing modules determines whether and when to initiate processing of second threads associated with writes in a second train during processing of the first threads.