Pairwise Combination Streaming Data Circulant Matrix Distribution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Constructing unique pairwise combinations from a stream of data elements is a computationally intensive task with O(n^2) time requirements, and existing approaches have seen only minor improvements in parallelization, often requiring significant memory and complex inter-processor communication.
Innovation Solution
Distributing data elements to processors based on the columns of a circulant matrix, allowing unique pairs to be constructed in a single pass with linear operations at each processor, reducing memory requirements and eliminating redundant operations, and using a master controller to manage data distribution and inter-processor communication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional parallelization approaches are used to construct pairwise combinations, then processing speed is improved, but memory requirements and inter-processor communication complexity increase significantly
Solution Approach 1:
The patent divides the data stream into multiple segments that are distributed across different processors. Each processor handles a specific segment independently, constructing pairwise combinations only within its assigned segment. This segmentation eliminates the need for complex inter-processor communication while maintaining parallel processing capabilities, directly resolving the contradiction between processing speed and communication complexity
Solution Approach 2:
The patent introduces a new dimensional organization of data using the circulant matrix structure, where data elements are arranged in a specific pattern that allows each processor to access only locally stored elements. This dimensional reorganization transforms the problem from requiring global data access to local data access, reducing inter-processor communication overhead while preserving parallel processing efficiency
2Productivity
If traditional parallelization approaches are used to construct pairwise combinations, then processing speed is improved, but memory requirements increase significantly
Solution Approach 1:
By segmenting the data stream and assigning specific segments to specific processors, each processor only needs to store and process its assigned segment in memory. This eliminates the need for all processors to hold the entire data set in memory simultaneously, reducing overall memory requirements while maintaining parallel processing speed through independent segment handling
Solution Approach 2:
Each processor performs pairwise combination operations only on its assigned data segment rather than on the complete data set. This partial action approach ensures that memory usage at each processor is proportional to its segment size rather than the total data size, significantly reducing memory requirements while the collective action of all processors maintains overall processing speed
3Productivity
If a single pass through the data stream is used to construct unique pairs, then processing efficiency is improved, but ensuring uniqueness of pairs becomes more difficult
Solution Approach 1:
The patent segments the data stream and assigns ordered segments to different processors. Each processor constructs pairs only from elements within its segment, and the global ordering is maintained through the segment assignment strategy. This segmentation approach allows single-pass processing while ensuring pair uniqueness through the structured distribution of data segments across processors
Solution Approach 2:
The data stream is pre-ordered and segments are assigned to processors in a specific sequence before processing begins. This preliminary organization of data ensures that when each processor independently constructs pairs from its segment, the resulting pairs are globally unique. The pre-established ordering and segment assignment prevent duplicate pairs while maintaining single-pass processing efficiency
Data Source
AI summary
A method is disclosed for pairwise combination of data elements of an input data stream. Steps are performed for each data element. A master controller reads the data from the stream and increments a counter. A register is instantiated, holding at least two words of memory and corresponding to the counter, by updating a register map and sending a message to a target processor via a logical pathway. The message instructs the target processor to create the register in local memory, including first and second memory locations, and an index corresponding to a value i. The target processor copies the data into the first memory location. For every other register corresponding to a value less than i, the other register copies the data into the second memory location and outputs a unique pair, constructed from the data elements stored at the first and second memory locations, to a database.