GPU-Based Parallel Similarity Calculation for Large-Scale Entity Correlation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing computing systems face significant computational intensity and time consumption when performing large-scale similarity calculations, particularly in retrieving the closest matches among millions of records, such as DNA samples, due to the need for extensive similarity measurements across large databases.
Innovation Solution
A system utilizing a graphics processing unit (GPU) to concurrently process pre-sorted data arrays using correlation algorithms like Jaccard similarity or conditional probability algorithms, allowing for parallel calculations and reducing the need for pre-sorting data, thereby accelerating similarity calculations and determining relationships among entities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional similarity calculation methods are used to retrieve top n matches from a database, then the process can be performed with standard processors, but the computational intensity and time consumption increase significantly as the number of stored records increases
Solution Approach 1:
The patent segments the similarity calculation task by dividing the database into multiple blocks and assigning different processing operations to different processors. The first processor handles data retrieval and pre-processing, while the second processor handles similarity calculations for specific blocks. This segmentation enables parallel processing and reduces the computational burden on any single processor, thereby increasing overall productivity without requiring excessive computational power for the entire dataset.
Solution Approach 2:
The patent introduces a multi-dimensional processing architecture that goes beyond conventional single-processor sequential processing. By utilizing multiple processors with different roles (first processor for data management, second processor for calculation) and organizing data into blocks, the system adds temporal and spatial dimensions to the calculation process. This dimensional expansion allows simultaneous operations that would be impossible on a single processor, dramatically increasing calculation speed while managing computational intensity through distributed architecture.
2Quantity of substance
If the number of stored records increases to provide more comprehensive data, then the completeness of search results improves, but the time required for similarity calculations increases
Solution Approach 1:
The patent segments the large database into multiple manageable blocks and distributes them across different processors. The first processor manages data retrieval from storage while the second processor performs similarity calculations on specific blocks in parallel. This segmentation allows the system to handle larger quantities of records without linearly increasing time consumption, as multiple calculations occur simultaneously across different blocks.
Solution Approach 2:
The first processor performs preliminary actions by retrieving and pre-processing data from storage before it is passed to the second processor for calculation. This preliminary data preparation and the block-based organization enable the similarity calculation processor to work more efficiently on pre-organized data, reducing the overall time required to process large datasets while maintaining comprehensive search coverage.
3Productivity
If data is pre-sorted to optimize similarity calculations, then the calculation efficiency improves, but the complexity of data preparation increases
Solution Approach 1:
The patent segments the data preparation task by having the first processor handle data retrieval and initial organization while the second processor handles the actual similarity calculations. This division of labor simplifies the complexity of data preparation, as only basic retrieval and blocking is required rather than complete pre-sorting, while still maintaining calculation efficiency through the structured block-based approach.
Solution Approach 2:
The patent changes the dimension of data organization from requiring fully pre-sorted data structures to using block-based partitions that can be processed in parallel. This dimensional shift in data organization allows the system to achieve calculation efficiency without the heavy complexity of comprehensive pre-sorting, as the block structure enables efficient parallel processing without requiring the data to be fully ordered beforehand.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for obtaining data stored at a storage device using a first processor of an entity correlation system. The data includes information about multiple entities. The first processor generates data arrays using the obtained data. Each data array includes parameter values for multiple entities and is configured for processing at a respective computing cell of a second processor. The system provides the data arrays to the second processor. The second processor is configured to execute a correlation algorithm to concurrently process the data arrays at the respective computing cells. The second processor computes a correlation score based on calculations performed at the cells using the algorithm and the parameter values. The system determines relationships among entities of the data arrays based on the correlation score. The relationships indicate overlapping attributes or similarities that exist among subsets of entities.