Dataset Intersection via Cache-Line Subset Transfer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current financial instrument trading systems face computational inefficiencies in determining dataset intersections due to the need for multiple memory transactions when comparing large datasets, leading to high system time and resource usage.
Innovation Solution
Organizing dataset items into subsets with unique attributes, allowing for minimal memory transactions by transferring only the necessary cache line-sized data between memory levels, and using techniques like hashing, masking, or sorting to determine subset membership efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional dataset intersection determination methods are used, then complete dataset comparison is achieved, but the number of memory transactions increases significantly
Solution Approach 1:
The patent divides the dataset into multiple subsets, where each subset contains items with the same subset attribute value. This segmentation allows the system to transfer only one cache line per subset attribute value instead of transferring entire datasets, dramatically reducing memory transactions while maintaining complete intersection determination accuracy
Solution Approach 2:
The patent performs preliminary organization of data into subsets with unique attributes before the intersection determination process. By pre-grouping items with identical subset attributes together and storing them in cache-friendly structures, the system prepares data in advance to minimize memory transactions during the actual intersection operation
2Measurement precision
If traditional dataset intersection determination methods are used, then all dataset items are compared, but the number of memory transactions increases
Solution Approach 1:
The patent extracts and utilizes the subset attribute from each dataset item to organize data into groups. By extracting this key attribute and using it to partition datasets, the system can work with smaller subset portions in cache memory rather than loading entire datasets, reducing memory transactions and computational resource usage while maintaining accurate intersection determination
3Quantity of substance
If large datasets are stored in memory, then complete intersection determination is possible, but memory transfer requirements increase
Solution Approach 1:
The patent introduces a new organizational dimension based on subset attributes, arranging dataset items in groups according to their attribute values. This dimensional reorganization allows the system to access and process data in cache-friendly patterns, transferring only necessary cache lines corresponding to each attribute value rather than transferring entire large datasets through memory hierarchy
Data Source
AI summary
An item is determined to exist in a dataset by arranging the dataset into a plurality of subsets, each bounded by the minimum amount of memory that may be transferred between levels of memory in a memory configuration. The item and the subsets have attributes that allow for a determination of which subset the item would exist in if the item were in the dataset. A singular subset is transferred between levels of memory to determine whether the item exists in the transferred subset. If the item does not exist in the transferred subset, it is determined that the item does not exist in the dataset.