Memory Sponge Spills Data to Peer Nodes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Shared-nothing query processing systems face performance degradation due to memory skew, leading to disk spilling, which is slow and inefficient, especially when processing large volumes of data, as mechanical disks are slower than solid-state components and can cause contention slowdowns.
Innovation Solution
Implementing a memory sponge that allows spilling to a peer node's primary memory instead of secondary memory, reducing the need for disk access and leveraging underutilized memory capacity across nodes for faster data handling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of stationary object
If data is spilled to secondary memory (disk), then memory capacity is extended, but processing speed decreases due to mechanical disk limitations
Solution Approach 1:
The patent introduces peer node primary memory as an intermediary storage location between local primary memory and secondary memory. When memory skew causes local primary memory to fill up, data is spilled to peer node's primary memory instead of immediately writing to slow mechanical disks. This intermediary solution provides faster access compared to disk while extending available memory capacity.
Solution Approach 2:
The patent creates copies of data in peer node primary memory (a form of volatile memory) instead of writing to secondary memory. This copying approach allows the system to maintain data in faster memory locations, effectively creating a distributed cache layer that speeds up processing while still providing memory extension.
2Quantity of substance
If disk spilling is used to handle memory overflow, then data can be stored, but system efficiency decreases due to contention slowdowns
Solution Approach 1:
The patent merges available primary memory resources across multiple peer nodes to create a shared memory pool. Instead of each node independently writing to its own slow secondary memory, the system combines memory resources node-to-node, effectively creating a distributed primary memory system that maintains high-speed access while providing expanded storage capacity.
Solution Approach 2:
The patent adds a new dimension to the memory hierarchy by utilizing the network dimension between nodes. Instead of the traditional single-node memory hierarchy (primary → secondary), the system creates a distributed hierarchy where peer node primary memory serves as an intermediate layer, effectively adding a spatial dimension to memory allocation.
3Productivity
If memory allocation is increased to handle large data groupings, then processing capability improves, but memory resource utilization becomes unbalanced
Solution Approach 1:
The patent makes peer node primary memory universally accessible to multiple nodes for spilling operations. Each node's primary memory can serve as overflow storage for multiple other nodes, creating a multi-functional memory resource that benefits the entire system. This universal access model improves overall memory utilization efficiency while maintaining high processing capability.
Solution Approach 2:
The system enables nodes to serve each other's memory needs through the spilling mechanism. When a node experiences memory pressure, it can utilize peer nodes' available primary memory without requiring centralized management or additional hardware resources. This self-service approach optimizes memory resource utilization across the distributed system.
Data Source
AI summary
Methods and systems are provided that may be utilized for spilling in query processing environments.