Parallel Sorting System with Periodic Data Dumping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Parallel sorting methods face inefficiencies due to peak CPU and I/O resource competition when distributing and sorting large volumes of data, leading to fluctuating CPU utilization and reduced throughput.
Innovation Solution
A data-driven parallel sorting method that distributes input data records circularly to multiple partitions, sorts records in parallel, and controls data dumping by writing sorted records into mass storage at calculated intervals, even when memory chunks are not full, creating a time difference in resource usage to avoid intense competition.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is distributed to multiple partitions for parallel sorting, then sorting efficiency is improved, but CPU and I/O resource competition peaks occur leading to fluctuating utilization and reduced throughput
Solution Approach 1:
The patent implements periodic action by controlling data dumping at calculated intervals (every └m/n┘ rounds) rather than continuously. This periodic dumping creates rhythmic I/O operations that avoid peak competition, as partitions are controlled to write to mass storage at staggered times rather than all simultaneously, thus maintaining sorting efficiency while reducing resource contention delays
Solution Approach 2:
The patent applies preliminary action by pre-calculating the optimal dumping interval (└m/n┘ rounds) before the sorting process begins. This pre-planned control strategy allows the system to proactively avoid resource competition peaks by scheduling data dumps in advance, rather than reacting to memory full conditions, thereby reducing delays while maintaining productivity
2Ease of operation
If data records are written into mass storage when memory chunks are full, then memory management is simplified, but I/O resource competition intensifies reducing throughput
Solution Approach 1:
The patent changes the parameter of dumping timing from event-driven (memory full) to interval-driven (every └m/n┘ rounds). This parameter transformation allows the system to maintain simple memory management logic while controlling I/O operations to occur at less congested times, thereby preserving ease of operation while improving throughput by avoiding I/O resource competition peaks
3Productivity
If all partitions sort data simultaneously, then parallel processing efficiency is maximized, but CPU resource competition increases causing utilization fluctuations
Solution Approach 1:
The patent applies periodic action to CPU resource utilization by controlling partitions to dump data at periodic intervals rather than continuously. This creates alternating phases of high CPU utilization (during sorting) and low CPU utilization (during controlled dumping), preventing sustained peak competition while maintaining overall parallel processing efficiency
Solution Approach 2:
The patent implements dynamics by making the dumping behavior adaptive and time-varying rather than static. Partitions dynamically adjust their dumping timing based on the calculated interval, creating a flexible system that maintains high parallel processing efficiency while naturally avoiding CPU resource competition peaks through temporal distribution of I/O operations
Data Source
AI summary
A data driven parallel sorting method includes distributing input data records to n partitions one by one in a circular manner. Each partition corresponds to a parallel sorting process with an allocated memory chunk sized to store m data records. The method also includes sorting, in parallel, current data records in respective memory chunks in respective partitions. The method also includes in response to distribution of data records of └m/n┘ rounds, circularly controlling one of the n partitions, and writing data records that have been sorted in the memory chunk of the partition into a mass storage as an ordered data chunk, and emptying the memory chunk. The method also includes in response to all data records being distributed, writing data chunks that have been sorted in respective memory chunks into the mass storage, and performing a merge sort on all ordered data chunks in the mass storage.