Concurrent Read Threads for Small File Backup Streaming
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data streaming methods for backup and migration operations face inefficiencies, particularly when dealing with multiple small files, leading to bottlenecks and discontinuous data streams, which can slow down the backup process and stress tape drives.
Innovation Solution
Implementing multiple concurrent thread instances for reading data from files, allowing each thread to handle data from a file and stream it into buffers, ensuring continuous and efficient data streaming by managing ownership and linking buffers appropriately.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple small files are read sequentially using a single thread, then the system structure remains simple, but the data stream becomes discontinuous and backup performance deteriorates
Solution Approach 1:
The patent divides the file reading operation into multiple concurrent threads, with each thread responsible for reading a specific file. This segmentation allows simultaneous reading of multiple small files, creating a continuous data stream that improves backup performance while managing complexity through structured thread organization.
Solution Approach 2:
The patent implements preliminary actions by opening and pre-loading file data into buffers before the actual backup write operation. Multiple threads perform file opening and data loading in advance, ensuring that data is ready for continuous streaming to tape without waiting for sequential file access.
2Speed
If a single thread reads data from files, then the implementation is straightforward, but bottlenecks occur and data streaming continuity is lost
Solution Approach 1:
The patent ensures continuity of useful action by having multiple threads read from multiple files simultaneously, preventing idle periods in the data stream. The buffering mechanism maintains continuous data flow to the tape drive, eliminating gaps that would occur with sequential single-threaded reading of small files.
Solution Approach 2:
The patent introduces buffers as an intermediary between the file reading threads and the tape writing operation. These buffers decouple the reading and writing processes, allowing multiple threads to feed data continuously while the tape write operation proceeds at its own pace, thus maintaining streaming speed without direct thread coordination overhead.
3Reliability
If data is streamed directly from disk to tape without buffers, then memory usage is reduced, but the tape drive experiences unnecessary acceleration and deceleration
Solution Approach 1:
The patent implements beforehand cushioning by using buffer memory to store data before writing to tape. This cushioning layer absorbs variations in data arrival rates from multiple threads and provides a steady data supply to the tape drive, preventing acceleration and deceleration cycles that would occur with direct disk-to-tape streaming.
Data Source
AI summary
Permitting multiple concurrent read operations upon a file system enhances small file performance. A respective instance of a thread routine stores the data from each file into a series of buffers for the file. Each instance is identified by a Process Identifier (PID) used for streaming the data from the buffers in the order in which the files are opened instead of the order in which the file data are returned by the read operations. In order to conserve buffer memory, read operations upon a file are suspended once a certain amount of data have been read from the file unless data from the file are being streamed from the buffers.