Shuffle Operator Memory Allocation for Scalable Query Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face limitations in processing speed due to hardware constraints, data storage methods, and restricted co-processing options, leading to inefficiencies in query execution.
Innovation Solution
A database system architecture that employs a parallelized data input, storage, and query processing system, utilizing a network of computing devices with independent processing core resources and optimized query plans to manage large datasets efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in a conventional sequential manner in database systems, then storage simplicity is maintained, but processing speed and query execution efficiency deteriorate due to hardware constraints and inability to parallelize operations
Solution Approach 1:
The patent divides data into multiple partitions distributed across different storage devices. Each partition can be accessed and processed independently by different computing resources, enabling parallel query execution and significantly improving processing speed while managing complexity through organized data segmentation
Solution Approach 2:
The patent transitions from sequential single-dimension data storage to a multi-dimensional distributed storage architecture across multiple devices and nodes. This dimensional expansion allows simultaneous access from multiple points, enhancing processing speed without proportionally increasing system complexity
2Productivity
If database systems use traditional single-node processing, then system simplicity is maintained, but query execution time increases and scalability is limited
Solution Approach 1:
The database system is segmented into multiple independent nodes, each capable of processing queries autonomously. Query operations are divided and distributed across these nodes, allowing parallel execution that improves productivity while maintaining manageable complexity through modular node design
Solution Approach 2:
Multiple independent database nodes are merged into a coordinated distributed system that functions as a unified database. This combination enables parallel processing and improved query execution efficiency while distributing complexity across individual nodes rather than concentrating it in a single system
3Productivity
If memory allocation is fixed in database systems, then system stability is maintained, but processing efficiency deteriorates when handling variable workload sizes
Solution Approach 1:
The patent implements dynamic memory allocation where memory resources are automatically adjusted based on workload demands. Memory units are allocated and deallocated dynamically during query execution, improving processing efficiency for variable workload sizes while managing complexity through automated memory management mechanisms
Data Source
AI summary
A node of a database system is operable to execute a shuffle operator based on, during a first temporal period, reserving a first amount of memory, allocating, to each of a plurality of other nodes, a corresponding number of fixed-memory units based on the first amount of memory, and process first data received from the plurality of other nodes in accordance with the corresponding number of fixed-memory units allocated to the each of the plurality of other nodes. During a second temporal period after the first temporal period, the first amount of memory is updated to a second amount of memory based on comparing an available amount of memory with a configured memory threshold, an updated corresponding number of fixed-memory units is re-allocated, and second data received from the plurality of other nodes in accordance with the corresponding number of fixed-memory units is processed.