Shared-Memory Hash Maps for Distributed Join Processing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing distributed processing systems face inefficiencies in performing join operations due to non-disjoint partitions of tables, necessitating data shuffling that increases overhead and hinders performance.

Innovation Solution

A distributed processing system utilizing a shared memory device to store a hash map of one table, allowing computing nodes to perform join operations directly with matching entries without requiring data shuffle between non-disjoint partitions.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If data is divided into non-disjoint partitions for distributed storage, then data distribution across computing nodes is achieved, but data shuffling overhead increases and join operation performance deteriorates

Engineering Contradiction:

Improvejoin operation performanceVSAvoiddata shuffling overhead

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

A shared memory device is introduced as an intermediary between storage and computing nodes. The target map is stored in this shared memory, allowing computing nodes to directly access target data without requiring data shuffling between non-disjoint partitions. This mediator eliminates the need for complex data redistribution while maintaining efficient join operations.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The target map is generated and stored in the shared memory device before the join operation begins. By preparing the target data structure in advance in a readily accessible location, the system avoids the need for runtime data shuffling and partitioning adjustments, thereby eliminating overhead and improving join performance.

Inventive Principle:

Principle #10Preliminary action

2Productivity

If target map is stored in shared memory device, then direct matching of entries is enabled, but memory resource consumption increases

Engineering Contradiction:

Improvejoin operation speedVSAvoidmemory resource consumption

Core Design Contradiction:

ProductivityVSQuantity of substance

Solution Approach 1:

Instead of storing the entire target table in shared memory, only the target map (a condensed representation containing only the essential indexing information) is stored. This localized storage approach provides the necessary functionality for fast joins while consuming minimal memory resources.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

A simplified copy of the target data (the target map) is created and stored in shared memory, rather than storing the complete target table. This copy contains only the essential information needed for join operations, enabling fast matching while significantly reducing memory consumption compared to storing the full dataset.

Inventive Principle:

Principle #26Copying

Data Source

PatentEP4610844A1Distributed processing system and method of operating the same

Publication Date: 2025.09.03 SK HYNIX INC

Figure 1
Figure 2
Figure 3

AI summary

Provided herein may be a distributed processing system and a method of operating the same. The distributed processing system may include a storage device, a plurality of computing nodes, and a shared memory device. The storage device may store a database including a plurality of tables. The plurality of computing nodes may generate a target map of the first table by applying a target function to a first table among the plurality of tables, and divide a second table among the plurality of tables into a plurality of partitions. The shared memory device may store the target map of the first table. The plurality of computing nodes may execute queries including a target operation using different partitions among the plurality of partitions of the second table and the target map.