Data Typing Probabilistic Maps Imbalanced Error Costs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data structures for determining type values of data keys consume excessive memory and processing power, especially when dealing with large datasets, and probabilistic structures trade accuracy for reduced memory at an unacceptable level.
Innovation Solution
A data structure is implemented that partitions data keys into groups based on query frequencies, using a combination of non-probabilistic and probabilistic data structures within each group, with more frequently encountered keys having higher accuracy and lower failure probabilities, and incorporating a least-recently used cache for expedited retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a straightforward implementation using an array or look-up table is used to store data key to type value associations, then type value retrieval accuracy is 100%, but memory consumption becomes prohibitively large and processing time increases significantly
Solution Approach 1:
The patent segments the data keys into multiple groups (tranches) based on their query frequencies. Frequently queried data keys are placed in separate groups from rarely queried keys, allowing different data structures to be used for each group. This segmentation enables the system to use more memory-intensive accurate structures only where needed, while using more space-efficient probabilistic structures for less critical data.
Solution Approach 2:
The patent applies different data structure qualities to different parts of the data. High-accuracy exact match structures are applied locally to frequently queried data keys, while probabilistic structures with lower accuracy are applied to rarely queried data keys. This local differentiation optimizes the balance between accuracy and memory consumption.
2Quantity of substance
If a probabilistic data structure is used to reduce memory footprint, then memory consumption decreases, but retrieval accuracy deteriorates to an unacceptable level
Solution Approach 1:
The patent segments data keys into frequency-based groups, allowing the system to apply probabilistic data structures only to the portions of data that can tolerate approximate matches (less frequently queried keys), while reserving exact match structures for frequently queried keys. This segmentation makes the memory savings from probabilistic structures meaningful while preserving accuracy where it matters most.
Solution Approach 2:
The patent changes the accuracy parameter of the data structure based on the query frequency of the data keys. For frequently queried keys, the system uses structures with high accuracy parameters; for rarely queried keys, it uses structures with lower accuracy parameters. This parameter adaptation allows the system to achieve acceptable overall performance while significantly reducing memory usage.
3Measurement precision
If all data keys are stored with high accuracy structures, then retrieval accuracy is maximized, but processing speed decreases due to larger data structure size
Solution Approach 1:
The patent segments data keys by query frequency into different tranches. Frequently queried keys are placed in tranches that can be quickly accessed, while rarely queried keys are placed in tranches that can use more space-efficient structures. This segmentation enables faster processing for the majority of queries while maintaining accuracy for critical data.
Solution Approach 2:
The patent applies high-performance data structures locally to frequently queried data keys, while using more space-efficient (but slower) structures for rarely queried keys. This local optimization ensures that the majority of queries, which target frequently accessed data, receive the fastest possible response times.
Data Source
AI summary
A plurality of data keys are associated with a plurality of type values; query frequencies of the data keys are known. A computer memory is divided into a plurality of tranches, each tranche including a probabilistic or non-probabilistic data structure. The data keys are stored in the tranches in accordance with their query frequencies such that, e.g., frequently queried data keys are stored in data structures having higher accuracy and infrequently queried keys are stored in data structure having less accuracy (and consequently require less memory space).