Cache-Aware Data Matching in Multiprocessor Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current approaches for identifying matches between data sets in multiprocessor systems are suboptimal, lacking efficiency and effectiveness in processing large data sets.
Innovation Solution
A cache-aware system and method that utilizes a multi-core processor to efficiently match data sets by partitioning data elements based on hash results, processing in sub-partitions, and utilizing hash tables to identify matches.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data sets are processed using traditional approaches in multiprocessor systems, then processing can be performed, but processing time is excessive and efficiency is low
Solution Approach 1:
The patent divides the data matching process into multiple partitions, with each partition handled by a different core or processing unit. Data elements are distributed across partitions based on hash results, enabling parallel processing and significantly reducing overall processing time while maintaining matching efficiency.
Solution Approach 2:
The system performs preliminary hashing of data elements before the actual matching process. By pre-computing hash results and organizing data into partitions based on these hashes, the system prepares the data structure in advance, allowing faster processing during the matching phase and reducing total execution time.
2Productivity
If all data elements are loaded into memory for processing, then matching can be performed, but memory consumption becomes excessive
Solution Approach 1:
The patent segments the data set into multiple partitions based on hash results, allowing only the necessary partition to be loaded into memory at any given time. This selective loading approach maintains fast matching speeds by keeping relevant data accessible while minimizing overall memory consumption through out-of-core processing.
Solution Approach 2:
The system uses hash tables and partition indexes as intermediary structures that reside in memory, while the actual data elements can be stored on disk. These intermediaries enable efficient lookup and matching operations without requiring all data to be simultaneously present in memory, thus reducing memory requirements while maintaining productivity.
3Productivity
If data elements are assigned to partitions based on hash results, then processing efficiency improves, but system complexity increases
Solution Approach 1:
The patent employs a universal hashing approach where the same hash function and partitioning logic can be applied to different data sets and processing scenarios. This multi-functionality allows the system to handle various matching tasks with a consistent framework, improving efficiency while managing complexity through reuse of proven components rather than creating specialized solutions for each case.
Data Source
AI summary
A system and method matches data from a first set of data with that of an other set of data in a manner based on the size of a cache.