Memory Analytics Partitioned Data Sampling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data processing systems face performance degradation due to the slower data retrieval from disk storage compared to Random Access Memory (RAM), especially when dealing with large amounts of data, leading to inefficient query execution and report generation.
Innovation Solution
A method is implemented in a data storage system that accesses only a portion of partitioned data based on probability calculations, determining the necessary amount to achieve a confidence threshold, thereby reducing the number of data accesses and improving query execution efficiency across multiple independent data partitions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in disk storage, then more storage capacity is available, but data retrieval speed deteriorates (100-1000 times slower than RAM)
Solution Approach 1:
The patent divides the data storage system into multiple independent data partitions distributed across different nodes. Each partition can be accessed independently and in parallel, enabling the system to process large datasets stored on disk with improved retrieval efficiency through concurrent access to multiple segments simultaneously.
Solution Approach 2:
The patent introduces a probabilistic sampling dimension by accessing only a portion of data items from each partition based on probability calculations. This selective access approach retrieves sufficient data to achieve confidence thresholds without reading entire partitions, effectively adding a dimension of selective data sampling that bypasses the slow disk retrieval bottleneck.
2Measurement precision
If all data items are accessed to ensure accuracy, then measurement precision is improved, but loss of time increases due to slower disk retrieval
Solution Approach 1:
The patent applies partial action by accessing only a calculated portion of data items from each partition rather than all items. The system determines the minimum number of items needed to achieve the desired confidence threshold, thereby reducing query execution time while maintaining sufficient accuracy for the query requirements.
Solution Approach 2:
The patent replaces the traditional mechanical approach of sequentially reading all data from disk with a probabilistic sampling mechanism. By using probability calculations to determine sample sizes and selecting items based on ranked characteristics, the system substitutes deterministic full-data access with a statistical approach that achieves comparable precision faster.
3Speed
If data is loaded into RAM for faster access, then data retrieval speed is improved, but use of energy increases and storage capacity is reduced
Solution Approach 1:
The patent segments the data access process into multiple independent partition accesses that can be performed in parallel. This allows the system to process large datasets stored on disk without loading them all into RAM simultaneously, reducing memory usage and associated energy consumption while maintaining fast retrieval through parallel processing of data segments.
Solution Approach 2:
The patent uses partial action by loading only a calculated portion of data items from disk into RAM based on probabilistic sampling requirements. Instead of loading entire partitions or large portions of data into memory, the system loads only the necessary sample items needed to achieve confidence thresholds, thereby reducing energy consumption and memory usage while still enabling fast in-memory processing of the selected data.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for memory analytics are disclosed. In one aspect, a method includes receiving input that requests a particular number of data items that are highest ranking in the data storage system based on a characteristic. The method includes accessing sets of data items to use in satisfying the request by selecting, from each of the independent data partitions, a set of data items that includes less than the particular number of data items that are highest ranking in the respective independent data partition based on the characteristic. The method includes comparing the sets of data items accessed from the plurality of independent data partitions. The method includes selecting the particular number of data items that are ranked highest based on the characteristic. The method includes providing, for output, the selected particular number of data items.