Adaptive Event Sampling for Real-Time Data Summarization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing insight services struggle to efficiently process vast amounts of file system events in computing environments, leading to impractical processing times for generating real-time data summarization due to the sheer volume of data, hindering the ability to dynamically generate evaluation outputs within a reasonable timeframe.
Innovation Solution
Adaptive sampling techniques are employed to selectively sample event records based on defined thresholds and criteria, using random numbers generated in memory to avoid disk access, allowing for the generation of evaluation outputs within acceptable timeframes while reducing computing resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all event records are processed to generate evaluation outputs, then measurement precision is improved, but processing time becomes impractical
Solution Approach 1:
The patent applies partial action by processing only a sampled subset of event records rather than all records. The sampling module selectively samples event records based on sampling parameters, achieving acceptable evaluation output accuracy while dramatically reducing processing time from impractical durations to acceptable timeframes.
Solution Approach 2:
The patent changes the parameter of data volume by using sampling to reduce the number of event records processed. By adjusting sampling parameters, the system dynamically controls the subset of data processed, balancing accuracy requirements against processing time constraints.
2Productivity
If sampling is used to reduce processing time, then productivity is improved, but measurement precision deteriorates
Solution Approach 1:
The system processes a partial subset of event records through sampling, achieving high processing speed while maintaining acceptable accuracy. The sampling module controls the subset size through configurable parameters, allowing the system to find the optimal balance between productivity and precision for different scenarios.
Solution Approach 2:
The sampling parameters are dynamic and can be adjusted based on system state, event types, and performance requirements. This allows the system to adapt the sampling rate to maintain precision when needed while maximizing productivity when event volumes are high or time constraints are loose.
3Productivity
If adaptive sampling is implemented, then productivity is improved, but device complexity increases
Solution Approach 1:
The system segments the event processing pipeline into distinct modules: a sampling module that selects event records based on sampling parameters, and an evaluation module that processes the sampled records. This segmentation allows each module to be optimized independently, improving overall productivity while managing complexity through modular design.
Solution Approach 2:
The sampling module serves multiple functions: it reduces data volume for faster processing, controls resource utilization, and can adapt to different event types and time periods through configurable parameters. This multi-functionality justifies the added complexity by providing versatile control over the evaluation process.
Data Source
AI summary
Techniques are provided for performing adaptive sampling for data summarization. An insight service may provide monitoring, troubleshooting, optimization, security, and/or other functionality for a computing environment. The insight service may intake millions to billions of events on a monthly basis from the computing environment, which are stored within a database. The insight service may provide data summarization for the events, which may include access patterns (e.g., file access patterns), anomalies, and ransomware detection. Dynamically querying and generating the data summarization may be impractical due to the sheer amount of events. Accordingly, adaptive sampling is provided for merely sampling certain events based upon various thresholds and criteria being met so that an evaluation output can be dynamically and efficiently generated within an acceptable time as the data summarization.