Distributed Processing Module Loading via On-Demand Code Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed processing systems face inefficiencies in data transfer volume and startup speed, particularly in master-slave configurations, where unnecessary data is transferred to slave nodes, leading to delays and redundant processing.
Innovation Solution
A distributed processing system that includes a module loader for initialization, a metadata management unit for acquiring initialization metadata, and a file management unit that checks for and loads execution code from other computers only when necessary, reducing data transfer and enabling high-speed processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If all modules are distributed to slave nodes in advance, then startup speed is improved, but data transfer volume increases significantly
Solution Approach 1:
The master node performs preliminary actions by distributing only metadata and module information to slave nodes before execution, rather than distributing all execution code. This allows slave nodes to prepare for startup without receiving unnecessary data, thus improving startup speed while reducing data transfer volume.
Solution Approach 2:
The patent extracts and separates metadata from execution code, distributing only the essential metadata to slave nodes in advance. The actual execution code is obtained on-demand during runtime, which reduces the initial data transfer volume while maintaining startup speed through selective data distribution.
2Productivity
If execution code is distributed to all slave nodes, then processing capability is improved, but communication overhead increases
Solution Approach 1:
The patent implements local quality by allowing different slave nodes to have different sets of modules distributed to them based on their specific processing needs. Each slave node receives only the metadata and execution code relevant to its assigned tasks, improving processing capability for specific functions while reducing overall communication overhead across the distributed system.
3Speed
If complete modules are transferred to slave nodes, then execution speed is improved, but network bandwidth consumption increases
Solution Approach 1:
The patent segments the module distribution process into two phases: metadata distribution (performed in advance to all slave nodes) and execution code distribution (performed on-demand to specific slave nodes). This segmentation allows execution speed to be maintained through pre-loaded metadata while significantly reducing network bandwidth consumption by avoiding transfer of complete modules to all nodes.
Data Source
AI summary
A distributed processing system in which a plurality of computers are interconnected, wherein each of the computers is provided with a module loader which loads each module and performs initialization processing, a metadata management unit which acquires metadata including a command for the initialization processing from a previously provided storage means or another computer, a file management unit which reads and writes a file within the storage means or the other computer, and an execution container which executes a distributed batch application. The file management unit examines whether or not an execution region including an execution code of a corresponding module is present in the storage means after the initialization processing, and when the execution region is not present, loads the execution code from the other computer and writes the loaded execution code as the execution region.