Spark Executor Thread Resource Release via External Shuffle Service
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing distributed storage method using Spark's external shuffle service is inefficient, leading to prolonged data transmission and resource wastage, as Spark continuously waits for a response after sending data, blocking subsequent tasks and utilizing resources unnecessarily.
Innovation Solution
Implementing a distributed storage method where the executor thread releases resources once data is sent to the external shuffle service, transitioning the task state to 'waiting-for-completion,' allowing the driver thread to reallocate resources without waiting for the task to be fully completed, thereby improving pipeline performance and reducing resource waste.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If Spark continuously waits for a response after sending data to the external shuffle service, then data storage reliability is ensured, but resource utilization deteriorates and task execution efficiency decreases
Solution Approach 1:
The patent segments the data storage process into two independent phases: (1) data transmission phase where Spark sends data to the external shuffle service, and (2) completion verification phase where the response is awaited. By separating these phases, the system allows subsequent tasks to proceed during the waiting period while maintaining reliability through eventual verification of storage completion.
Solution Approach 2:
The patent implements preliminary action by allowing Spark to send data to the external shuffle service and immediately proceed with subsequent tasks without waiting for the storage completion response. The system performs the data transmission action first, then handles the response asynchronously, enabling overlapping of I/O operations with computation operations.
2Ease of operation
If Spark waits for the external shuffle service response before allocating new tasks, then resource allocation control is maintained, but resource utilization deteriorates due to idle executor threads
Solution Approach 1:
The patent implements feedback mechanisms through callbacks that notify Spark when data storage is complete. The external shuffle service provides feedback about storage status, allowing the system to dynamically adjust resource allocation based on actual storage completion rather than waiting passively. This maintains control while improving resource utilization.
Solution Approach 2:
The patent ensures continuity of useful action by keeping executor threads active and allocating new tasks continuously without idle waiting periods. The system maintains continuous productive work by overlapping data transmission with subsequent task execution, eliminating the idle periods that previously occurred while waiting for storage responses.
3Manufacturing precision
If the external shuffle service merges and sorts data before storage, then data organization quality is improved, but processing time increases and pipeline performance deteriorates
Solution Approach 1:
The patent applies dynamics by making the data processing pipeline flexible and adaptive. Instead of rigid sequential processing where merging and sorting must complete before storage begins, the system allows overlapping of operations. The external shuffle service can begin merging and sorting data while Spark simultaneously sends new data batches, creating a dynamic pipeline that improves throughput.
Solution Approach 2:
The patent implements preliminary action by allowing the external shuffle service to start merging and sorting data in advance while Spark continues sending data. The service performs preliminary data organization operations concurrently with data reception, preparing data for storage without blocking the overall pipeline and reducing total processing time.
Data Source
Figure 1~2
Figure 3
Figure 4~5
AI summary
The present disclosure provides a distributed storage method, involving the technical fields of computer and cloud computing, and including: reading and sending (201) data to an external shuffle service in response to a request of a task from a driver thread; modifying (202) a state of the task to a waiting-for-completion state after finishing sending the data to the external shuffle service; and sending (203) the waiting-for-completion state to the driver thread, to cause the driver thread to release an executor thread corresponding to the task. The distributed storage method can reduce the waste of the resources of the executor thread and improves the efficiency of task operations. The present disclosure also provides a distributed storage device, an electronic apparatus, and a computer-readable medium.