Mirrored File System for Distributed Build Speed
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for distributing the build of a code base are limited by CPU and I/O intensive pre-processing of source code files, which consumes processing resources and results in suboptimal parallel processing performance when using traditional shared file systems.
Innovation Solution
Implementing a mirrored file system on volunteer machines, where inputs are locally retrieved and reused if consistent with the recruiter machine's file system, and inputs not cached are obtained from the recruiter machine, allowing for efficient distribution of data processing tasks and improved build speeds through parallel processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If pre-processing of source code files is performed at the recruiter machine, then the code base can be built, but CPU and I/O resources are consumed and the number of volunteer machines that can be recruited is limited
Solution Approach 1:
The patent divides the file system into multiple mirrors distributed across different volunteer machines. Each mirror contains a portion of the source code files, allowing the recruiter machine to distribute pre-processing tasks to multiple volunteer machines simultaneously. This segmentation enables parallel pre-processing without concentrating all CPU and I/O loads on a single recruiter machine.
Solution Approach 2:
The patent introduces a distributed file system architecture that adds a spatial dimension to file storage and access. Instead of a single centralized file system, multiple mirrored copies are distributed across different machines, allowing volunteer machines to access files locally or from other mirrors, thereby reducing the I/O burden on the recruiter machine and enabling recruitment of more volunteer machines.
2Productivity
If traditional shared file systems are used for distributing builds, then volunteer machines can access source code files, but the file system operations are slow and parallel processing performance is suboptimal
Solution Approach 1:
The patent segments the file system into multiple independent mirrors distributed across different volunteer machines. Each mirror can be accessed independently, allowing multiple volunteer machines to perform file operations in parallel without contending for access to a single shared file system. This eliminates the bottleneck of traditional shared file systems and enables optimal parallel processing performance.
Solution Approach 2:
The patent creates multiple copies (mirrors) of the file system and distributes them across different volunteer machines. Each volunteer machine can access files from its local mirror or from other mirrors, eliminating the need for slow network file operations over a traditional shared file system. This copying approach maintains data consistency while dramatically improving file access speed for parallel processing.
3Adaptability or versatility
If all source code files are pre-processed at the recruiter machine, then compilation can be distributed, but the recruiter machine's processing resources are exhausted limiting the number of recruitable volunteer machines
Solution Approach 1:
The patent segments the pre-processing workload by distributing source code files across multiple mirrors on different volunteer machines. The recruiter machine only needs to coordinate this distribution and handle metadata operations, while actual file access and local pre-processing occur on volunteer machines themselves. This enables the recruiter machine to coordinate many more volunteer machines without being constrained by its own processing power.
Solution Approach 2:
The patent enables volunteer machines to serve themselves by providing local mirrors of source code files. Each volunteer machine can access files from its local mirror without requiring the recruiter machine to provide continuous processing support. This self-service capability allows the recruiter machine to recruit and coordinate a large number of volunteer machines while maintaining its processing resources for coordination tasks.
Data Source
AI summary
Mounting and populating a mirrored version of at least a portion of a file system of a recruiter machine at a volunteer machine performing one or more data processing tasks for the recruiter machine is disclosed. In some embodiments, an input stored in such a dedicated file system for the recruiter machine at the volunteer machine may be locally retrieved and reused, if still consistent with a corresponding input stored in a file system at the recruiter machine, when performing a data processing task for the recruiter machine at the volunteer machine. In some embodiments, if an input required by the volunteer machine to perform a data processing task for the recruiter machine is not cached in such a dedicated file system for the recruiter machine at the volunteer machine in a state that is consistent with a corresponding input stored in a file system at the recruiter machine, the input is obtained from the recruiter machine.