Incremental Dataset Sorting with Overlapped I/O and Compute
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing methods for sorting large datasets are hindered by the significant time required for input/output operations, with CPU computation time being overshadowed by I/O delays, especially in systems where I/O has become faster, necessitating optimization of the sorting process.
Innovation Solution
The system splits unsorted input data into smaller subsets and sorts each subset concurrently with reading the next subset, using a merge sort approach that overlaps computation with I/O operations by performing a merge while writing previous subsets, employing a quick sort algorithm to optimize cache performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If data is sorted sequentially after complete input, then sorting accuracy is ensured, but total sorting time increases due to sequential processing
Solution Approach 1:
The patent applies preliminary action by performing sorting operations on data subsets before the complete dataset is available. Specifically, it sorts the first subset of data immediately after receiving it, while simultaneously receiving the second subset of data. This overlapping of reception and sorting operations reduces the total time required while maintaining sorting accuracy through subsequent merge operations.
2Reliability
If I/O operations are performed sequentially, then data integrity is maintained, but CPU utilization decreases due to waiting time
Solution Approach 1:
The patent implements continuity of useful action by ensuring the CPU continuously performs sorting operations without idle waiting time. While the first data subset is being sorted, the system simultaneously receives the second data subset. This eliminates CPU idle time and maintains continuous productive work, thereby improving CPU utilization while preserving data integrity through structured merge operations.
3Manufacturing precision
If complete data is read before sorting, then sorting completeness is ensured, but I/O time dominates the total processing time
Solution Approach 1:
The patent applies segmentation by dividing the complete dataset into multiple subsets (first subset, second subset, etc.) and processing them separately but concurrently. The system receives and sorts the first subset while simultaneously receiving the second subset, and continues this pattern. This segmentation allows sorting operations to overlap with I/O operations, reducing the overall time duration while ensuring sorting completeness through subsequent merge operations that combine all subsets.
Data Source
AI summary
A method of sorting a dataset includes incrementally receiving data from the dataset, and incrementally storing the received data as individual input data subsets as the data is received, thereby sequentially generating a plurality of filled data subsets of unsorted data. The method includes individually sorting each filled data subset of unsorted data concurrently with receiving data for a next one of the individual input data subsets, thereby sequentially generating a plurality of sorted input data subsets, and performing a merge sort on the plurality of sorted input data subsets, thereby incrementally generating a sorted version of the dataset.