File Archiving System Thread Contention Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing file archiving systems face inefficiencies due to thread contention when processing files for archiving policies, leading to slower processing times and increased management overhead, especially when dealing with unevenly distributed files across folders and competing threads accessing shared lists.
Innovation Solution
A method employing a single input reader thread to scan folders and generate lists of files, with multiple map threads processing files through filters based on archiving policies, and reduce threads grouping files by key values to eliminate thread contention and enhance processing efficiency, using a map-reduce programming model to manage file archiving operations such as move, purge, index, and delete.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple threads process files simultaneously using shared lists, then processing throughput increases, but thread contention increases causing slower processing times
Solution Approach 1:
The patent divides the file processing workload into separate segments by creating dedicated lists for different archiving actions (move list, purge list, index list, delete list). Each map thread processes files independently and adds them to the appropriate list based on policy filters, eliminating thread contention by avoiding shared data structures.
Solution Approach 2:
The patent introduces reduce threads as intermediaries between map threads and the final processing stage. These reduce threads collect files from the various action lists and prepare them for execution, acting as a buffer that decouples the map threads from each other and preventing direct contention.
2Productivity
If multiple threads access shared lists simultaneously, then file processing can be parallelized, but management overhead increases
Solution Approach 1:
The patent segments the processing architecture into distinct map threads and reduce threads with clearly defined responsibilities. Map threads generate files for archiving actions and add them to dedicated lists, while reduce threads consume from these lists and prepare files for execution. This segmentation eliminates the need for complex synchronization mechanisms.
Solution Approach 2:
The patent extracts the list management functionality from the map threads and places it in dedicated reduce threads. This extraction allows map threads to focus solely on processing files through policy filters without the burden of managing shared data structures, significantly reducing management overhead.
3Reliability
If files are unevenly distributed across folders, then scanning all folders takes longer, but comprehensive archiving policy application is required
Solution Approach 1:
The patent segments the folder scanning and file processing operations across multiple map threads that work in parallel. Each map thread processes a portion of the files generated by the input reader, applying policy filters independently. This parallel processing approach ensures comprehensive policy application while reducing total scanning time despite uneven file distribution.
Data Source
AI summary
Methods and apparatus for archiving files in a system with primary storage and secondary storage that reduces thread contention. A single input reader thread generates list of files for processing by multiple map threads that evaluate each file in a list against a number of archive policies, such as move, purge, index and delete. Reduce threads take the lists from the map threads and generates list of files for action under a given policy, e.g., move, purge, index, delete. This arrangement eliminates thread contention for files and lists.