Shared-Memory Target Maps for Shuffle-Free Distributed Joins

Solution Overview

Problem

Existing distributed processing systems face inefficiencies in performing join operations due to the need for data shuffling when table partitions are not disjoint, leading to unnecessary overhead and reduced performance.

Innovation Solution

A distributed processing system utilizing a shared memory device to store a hash map of one table, allowing computing nodes to perform join operations directly with matching entries without requiring data shuffle, by using a hash function to generate disjoint partitions.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If data shuffling is performed to ensure disjoint partitions for join operations, then join operation correctness is improved, but processing time and system overhead increase

Engineering Contradiction:

Improvejoin operation correctnessVSAvoidprocessing time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs preliminary actions by pre-computing and storing target maps in shared memory before join operations are executed. These target maps contain pre-calculated mapping relationships that enable direct access to matching partitions without requiring data shuffling during the actual join operation, thus eliminating the time-consuming shuffle step while ensuring correctness

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent introduces target maps as an intermediary data structure that mediates between the distributed table partitions and the join operation. These target maps stored in shared memory serve as a lookup mechanism that directly identifies matching partitions without requiring data movement, acting as a mediator that eliminates the need for traditional data shuffling while maintaining join correctness

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If data shuffling is performed to handle non-disjoint table partitions, then join operation accuracy is improved, but computational overhead increases

Engineering Contradiction:

Improvejoin operation accuracyVSAvoidcomputational overhead

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The system performs preliminary actions by pre-computing and storing target maps in shared memory before join operations are executed. These target maps contain pre-calculated mapping relationships that enable direct access to matching partitions without requiring data shuffling during the actual join operation, thus eliminating the time-consuming shuffle step while ensuring correctness

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent introduces target maps as an intermediary data structure that mediates between the distributed table partitions and the join operation. These target maps stored in shared memory serve as a lookup mechanism that directly identifies matching partitions without requiring data movement, acting as a mediator that eliminates the need for traditional data shuffling while maintaining join correctness

Inventive Principle:

Principle #24Intermediary (Mediator)

3Device complexity

If traditional distributed join processing is used without shared memory, then system simplicity is maintained, but join operation performance decreases

Engineering Contradiction:

Improvesystem simplicityVSAvoidjoin operation performance

Core Design Contradiction:

Device complexityVSProductivity

Solution Approach 1:

The shared memory device serves multiple functions: it stores target maps for efficient lookup, enables direct access to matching partitions, and eliminates the need for data shuffling. This multi-functional component improves join operation performance while maintaining relative system simplicity by providing a centralized storage solution that benefits the entire distributed processing system

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The patent introduces a new dimension to the distributed processing architecture by adding shared memory as a centralized resource that all computing nodes can access. This dimensional addition allows for O(1) lookup of matching partitions through target maps, dramatically improving join performance without fundamentally complicating the distributed processing model

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

Data Source

PatentUS20250272294A1Distributed processing system and method of operating the same

Publication Date: 2025.08.28 SK HYNIX INC

AI summary

Provided herein may be a distributed processing system and a method of operating the same. The distributed processing system may include a storage device, a plurality of computing nodes, and a shared memory device. The storage device may store a database including a plurality of tables. The plurality of computing nodes may generate a target map of the first table by applying a target function to a first table among the plurality of tables, and divide a second table among the plurality of tables into a plurality of partitions. The shared memory device may store the target map of the first table. The plurality of computing nodes may execute queries including a target operation using different partitions among the plurality of partitions of the second table and the target map.