Running Profiles for Real-Time Event Stream Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current techniques lack effective methods for real-time analysis of event data streams, particularly in identifying overall trends and irregularities across groups of entities, as accessing historical data is slow and existing methods fail to accurately compare individual data streams against peer groups in real-time.
Innovation Solution
A system and method that utilizes running profiles and trend groups to analyze event data in real-time by generating and maintaining normalized running profiles for both individual entities and trend groups, allowing for the comparison of individual data streams against group profiles to detect irregularities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If historical data is accessed to identify trends and patterns, then measurement precision is improved, but processing speed deteriorates
Solution Approach 1:
The system pre-calculates and maintains running statistical summaries (mean, standard deviation, percentiles) for historical data in memory structures. This preliminary processing allows the system to quickly query historical patterns without accessing raw historical data during real-time analysis, thus maintaining both accuracy and speed.
Solution Approach 2:
The invention extracts key statistical characteristics from historical data and stores them as condensed summaries rather than the full historical data itself. This extraction process separates the essential trend information from the bulk historical data, enabling fast retrieval of trends while minimizing data access requirements.
2Speed
If running estimates are used to provide statistical summaries, then processing speed is improved, but measurement precision deteriorates
Solution Approach 1:
The system continuously updates running statistical summaries with new data events and uses these updated summaries to compare against historical patterns. This feedback loop ensures that the running estimates remain accurate and adaptive, maintaining measurement precision while achieving real-time processing speeds.
Solution Approach 2:
The invention combines multiple statistical metrics (mean, standard deviation, percentiles) into a composite running profile for each entity. This composite approach integrates diverse statistical information into a unified structure that provides both computational efficiency and comprehensive analytical accuracy.
3Measurement precision
If individual data streams are compared against peer groups, then detection accuracy is improved, but device complexity increases
Solution Approach 1:
The system segments the analysis into independent components: entity-level running profiles, group-level running profiles, and comparison logic. Each component processes specific data streams or performs specific comparisons independently, making the overall complex task of peer-group analysis manageable and scalable through modular architecture.
Solution Approach 2:
The invention implements a universal running profile structure that can be applied to any entity within a group. The same profile generation and comparison algorithms work across all entities and groups, reducing the need for entity-specific complex logic and simplifying the overall system architecture.
Data Source
AI summary
A system. method and program product that utilizes running profiles and trend groups to analyze streams of event data in real time. A system is provided having a group profile processing system for generating a running group profile from a plurality of data event values, wherein each of the plurality of data event values is obtained from an associated one of the plurality of data streams; an event normalization system for normalizing each of the plurality of data event values based on the running group profile to generate a normalized data event value for the data event values obtained from each of the plurality of data streams; an event profile processing system for generating a running event profile for each normalized data event value; and an analysis system that evaluates each data stream by comparing a normalized data event value with an associated running event profile.