Coordinator Server Buffer Control for Parallel Database Pipeline Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In parallel database systems, the coordinator server often struggles to keep pace with the processing speed of database servers, leading to data accumulation and resource pressure, especially during pipeline processing, which can result in decreased responsiveness and increased overhead.
Innovation Solution
The implementation of a coordinator server with a buffer unit for integrating data sets from database servers, a transmitting unit for sending merged data to clients, and a control unit for managing data transmission based on data size, along with database servers that adjust their transmission block size dynamically in response to control commands, ensuring efficient data transfer and resource management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If pipeline processing is implemented in parallel database systems, then processing efficiency is improved, but data accumulates on the coordinator server causing resource pressure
Solution Approach 1:
A buffer management unit is introduced as an intermediary between database servers and the coordinator server. This buffer unit temporarily stores data transmitted from database servers and controls the flow of data to the coordinator server, preventing data accumulation while maintaining pipeline processing efficiency.
Solution Approach 2:
The buffer management unit monitors the amount of data in the buffer and dynamically adjusts the transmission block size of database servers based on buffer status. When the buffer is nearly full, transmission is slowed; when buffer space is available, transmission is accelerated, creating a feedback control mechanism that balances throughput and resource usage.
2Ease of operation
If fixed transmission block size is used, then implementation is simple, but initial responsiveness is poor and overhead increases
Solution Approach 1:
The transmission block size is changed from a fixed value to a dynamic value that adjusts based on buffer status. The buffer management unit controls the transmission block size to increase or decrease according to available buffer space, improving responsiveness and reducing overhead while maintaining implementation feasibility through centralized control.
3Productivity
If transmission block size is increased, then throughput is improved, but resource pressure on coordinator server increases
Solution Approach 1:
The buffer acts as an intermediary that decouples the transmission rate from the processing rate at the coordinator server. By controlling the transmission block size based on buffer status, the system can achieve high throughput during periods of available buffer space while preventing resource pressure by slowing transmission when the buffer is full.
Data Source
AI summary
A first transmitting unit transmits a processing command to a plurality of parallelized database servers. A second transmitting unit integrates data sets transmitted from the database servers in response to the processing command, and transmits an integrated data set to a client. An integrating unit integrates data sets buffered in a buffer unit. A determining unit determines a transmission start or a transmission suspend of the data sets based on a data size in the buffer unit. A third transmitting unit transmits a control command for the transmission start or the transmission suspend to the database servers based on a result of determination by the first determining unit.