Nested probabilistic data structures for big data analytics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current big data analytics in cloud computing environments face substantial storage costs and query serving times due to the need to process large volumes of data across complete storage systems, particularly in domains like API management where storing and retrieving data records for extended periods is time-consuming and costly.
Innovation Solution
The implementation of a nested probabilistic data structure, such as the count-min sketch, which uses hash functions to efficiently store and retrieve data by creating multiple levels of nesting and employing probabilistic methods to provide fast and accurate query responses, reducing storage needs and query complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Duration of action of stationary object
If traditional columnar storage is used to store big data for extended periods, then data retention capability is improved, but storage costs increase substantially
Solution Approach 1:
The patent implements a nested probabilistic data structure where multiple Count-Min Sketches are organized in nested levels. Each level handles a specific time window (e.g., hourly, daily, monthly), allowing the system to retain data for extended periods while maintaining compact storage. The nested structure enables efficient data retention without requiring proportional storage space for the entire retention period.
Solution Approach 2:
The system changes the storage parameter from storing all historical data to storing only probabilistic summaries (Counts) within specific time windows. By parameterizing the retention strategy to keep only the most recent N time windows at each nesting level, the system achieves extended retention capability with significantly reduced storage requirements.
2Productivity
If traditional distributed frameworks like MapReduce are used to perform analytics, then processing capability is improved, but query serving time increases substantially
Solution Approach 1:
The system performs preliminary actions by continuously maintaining probabilistic summaries of data in nested Count-Min-Sketches as data streams in. This pre-computed summary structure allows queries to be answered by simple lookups and aggregations of existing summaries rather than reprocessing the entire historical data, dramatically reducing query serving time while maintaining processing capability.
Solution Approach 2:
Instead of copying and processing all historical data for each query, the system uses probabilistic summaries (Copies of aggregated information) that represent the data distribution. These summaries can be quickly aggregated across nested levels to answer queries about different time windows without accessing the original data, reducing query time while preserving analytical accuracy.
3Measurement precision
If database engines process all records across complete storage for queries, then query accuracy is improved, but processing time increases substantially
Solution Approach 1:
The patent applies partial action by processing only the necessary portions of data through probabilistic summaries rather than all records. The Count-Min-Sketch structures provide approximate counts that are sufficient for analytics queries, avoiding the need to scan and process every historical record while maintaining adequate query accuracy for business intelligence purposes.
Data Source
AI summary
A system may include a nested probabilistic data structure that contains nested tables, each table having electronic records associated with different hash functions. A data analytics engine may receive information about a detected action, including action characteristics. In response to the detected action, insertions are executed into the nested probabilistic data structure at multiple levels based on the action characteristics (each insertion incrementing an entry in the data structure selected using the hash functions). The engine then continues to monitor for detected actions. Upon receipt of a query associated with the monitored action (including a query type), a key and level of nesting associated with the query is determined based on the query type. The hash functions are executed on the determined key to retrieve a value from each electronic record in an appropriate nested table, and a minimum of the retrieved values is output in response to the query.