Real-Time Aggregation Primitives for Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data processing systems face challenges in achieving real-time response times and thorough analysis of vast, distributed data stores, particularly in big data analytics, with inefficiencies in response time, accuracy, and flexibility.
Innovation Solution
A data processing system that defines a set of metrics and aggregation primitives, calculating and storing these primitives in real-time without storing raw data, allowing for fast and flexible metric calculation upon request, and enabling the reuse of aggregation primitives for various metrics and new queries without code recompilation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If raw data is stored and processed in real-time, then response time is improved, but storage requirements and processing load increase significantly
Solution Approach 1:
The patent extracts only the necessary aggregated information from raw data and stores only that in the data store. Instead of storing all raw interaction data, the system stores pre-computed aggregation primitives (e.g., count, sum, average) that can be queried to obtain required metrics, thereby reducing storage requirements while maintaining fast response times.
Solution Approach 2:
The system performs preliminary aggregation of data into primitives in real-time as data arrives, so that when queries are needed, the aggregated data is already prepared and available. This preliminary processing eliminates the need to reprocess raw data for each query, reducing both storage needs and query time.
2Measurement precision
If thorough analysis of distributed data is performed, then accuracy is improved, but processing time increases
Solution Approach 1:
The patent segments the data analysis into two stages: (1) real-time aggregation of raw data into primitive statistics as data arrives, and (2) query-time computation of specific metrics from the pre-aggregated primitives. This segmentation allows thorough analysis to be performed on manageable data chunks rather than entire distributed datasets, improving accuracy while reducing processing time.
Solution Approach 2:
The system introduces aggregation primitives as intermediary representations between raw data and final metrics. These primitives serve as a mediator that summarizes raw data in a compact form, enabling accurate analysis without requiring processing of the full raw dataset for each query.
3Adaptability or versatility
If data processing system stores all raw data, then flexibility for new queries is improved, but device complexity and storage costs increase
Solution Approach 1:
The patent creates a universal query interface that can answer diverse metric queries by combining a small set of pre-computed aggregation primitives. The same primitives (e.g., count, sum, average) can be combined in different ways to satisfy various query requirements, providing flexibility without requiring storage or processing of all raw data for each possible query type.
Data Source
AI summary
A data processing system for processing data related to incoming interactions. The data processing system comprises a first configuration object defining a set of metrics; a second configuration object defining a number of aggregation primitives, by analyzing requirements of the set of metrics; a first processing unit adapted for receiving data related to incoming interactions, and adapted for real time calculation of at least one aggregation primitive based on this data taking into account the second configuration object; a data storage element for storing the calculated at least one aggregation primitive without storing raw data related to the incoming interactions; a second processing unit adapted for calculating one or more metrics based on the stored at least one aggregation primitive taking into account the first and the second configuration object.