Parallel Distinct Operations on Shared Memory Tables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In parallel computing environments with shared memory, existing technologies face challenges in efficiently performing relational distinct operations across multiple tables, particularly due to the difficulty in parallelizing operations originally designed for single-core or single-chip systems, as processor frequency increases are limited, and software must adapt to multi-core processors.
Innovation Solution
A computer-implemented method and system that executes distinct operations on multiple tables by using a parallel hash table and hash map to eliminate duplicates, employing a hierarchy of table pairs and interdistinct operations, which reduces the row set and detects duplicates through a series of algorithms involving group by operations and perfect hash maps.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If distinct operations are performed on multiple tables in parallel computing environments, then productivity is improved, but device complexity increases due to the need for parallelization algorithms and shared memory management
Solution Approach 1:
The distinct operation on multiple tables is divided into separate distinct operations on individual tables, executed in parallel by different processor cores. Each core independently processes one table, eliminating the need for complex inter-core communication and synchronization during the distinct operation itself.
Solution Approach 2:
A shared memory structure serves as an intermediary where distinct results from multiple tables are stored. The system uses a centralized coordination mechanism that manages the parallel execution and collects results from multiple processor cores without requiring complex point-to-point communication between cores.
2Productivity
If processor frequency is increased to improve computation power, then productivity is improved, but heat production and signal propagation delay increase due to physical limits
Solution Approach 1:
The system dynamically utilizes multiple processor cores that can be activated independently, allowing computation power to scale with the number of cores rather than relying on increasing the frequency of a single core. This dynamic parallel execution avoids the thermal constraints of high-frequency single-core operation.
3Productivity
If distinct operations are parallelized across multiple processor cores, then productivity is improved, but difficulty of detecting and measuring increases due to duplicate detection across reduced row sets
Solution Approach 1:
The system extracts and eliminates duplicates within each table's reduced row set independently before combining results. By performing duplicate detection and elimination at the individual table level rather than across all tables simultaneously, the complexity of duplicate detection is significantly reduced while maintaining parallel processing benefits.
Data Source
AI summary
A computer-implemented system and method for performing distinct operations on multiple tables of shared memory of parallel computing environments are disclosed. A distinct operation is executed on each table of a plurality of tables, each distinct operation eliminating duplicate data from each table, the executing creating a hierarchy of table pairs and distinct results, the distinct results comprising a reduced row set for each table. Duplicates on each reduced row set are detected to complete the distinct operation on the plurality of tables.