Bounded Group By Query System for Security Event Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional database systems face challenges in efficiently computing group statistics for log data, as the number of groups is often unknown, leading to unconstrained memory use and query failures.
Innovation Solution
A bounded group by query system that computes approximate time-sliced statistics by performing a single pass scan of the dataset, using sized-bounded data structures, and distributing query execution across accumulator nodes and an aggregator node.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional database systems perform group by queries on log data, then group statistics can be computed, but the number of groups is unknown leading to unconstrained memory use and query failures
Solution Approach 1:
The patent changes the parameter of data structure sizing from unbounded to bounded by introducing a fixed maximum size parameter for the hash table. This allows the system to compute group statistics with controlled memory consumption, resolving the contradiction between measurement precision and quantity of substance.
Solution Approach 2:
The patent applies partial action by computing statistics only for groups that fit within the bounded data structure capacity. When the maximum size is reached, the system stops adding new groups and continues processing, thereby providing partial but useful statistical information within constrained memory resources.
2Device complexity
If an initial pass of log data is performed to determine distinct groups, then the number of groups can be determined, but the process becomes extremely time consuming
Solution Approach 1:
The patent performs preliminary action by pre-allocating a bounded data structure with a fixed maximum size before processing the log data. This eliminates the need for an initial scanning pass to determine the number of groups, as the system is already prepared with sufficient capacity to handle the data within constrained memory resources.
Solution Approach 2:
The patent segments the group statistics computation into two parts: groups that fit within the bounded data structure are tracked exactly, while groups exceeding the boundary are handled differently. This segmentation allows the system to avoid time-consuming initial scans while still providing useful statistical information.
3Quantity of substance
If sized-bounded data structures are used to store group statistics, then memory use is constrained, but approximate results are produced instead of exact results
Solution Approach 1:
The patent changes the parameter of result precision by introducing a bounded data structure that trades off exactness for memory efficiency. The system configures a maximum size parameter that controls both memory consumption and the level of approximation in the results, allowing flexible adjustment based on resource constraints.
Solution Approach 2:
The patent applies discarding by not tracking groups that exceed the bounded data structure capacity. Instead of storing all groups exactly, the system discards overflow groups and recovers memory space, producing approximate results that reflect the most significant groups while consuming constrained memory resources.
Data Source
AI summary
Systems and methods are disclosed to implement a bounded group by query system that computes approximate time-sliced statistics for groups of records in a dataset according to a group by query. In embodiments, a single pass scan of the dataset is performed to accumulate exact results for a maximum number of groups in a result grouping structure (RGS) and approximate results for additional groups in an approximate result grouping structure (ARGS). RGSs and ARGSs are accumulated by a set of accumulator nodes and provided to an aggregator node, which combines the received structures to generate exact or approximate statistical results for at least a subset of the groups in the dataset. Advantageously, the disclosed query system is able to produce approximate results for at least some of the groups in a single pass of the dataset using size-bounded data structures, without predetermining the actual number of groups in the dataset.