Near Real-Time Data Aggregation via Pre-Computed Event Records
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing batch applications face latency issues due to the aggregation of large volumes of data, particularly in real-time data processing, as they rely on retrieving data from data warehouses, which hampers efficient execution of applications across different data streams.
Innovation Solution
A method that intermittently receives data from multiple streams, detects specific records, creates collections, searches for historical aggregations, computes combined data, and applies rules to generate a wide record with real-time aggregations, enabling near real-time processing and reducing latency by eliminating the need for database queries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If batch applications retrieve data from data warehouse to compute aggregations, then data aggregation can be performed, but latency increases and real-time processing capability deteriorates
Solution Approach 1:
The system pre-computes and stores aggregations in the data warehouse before they are needed. When real-time queries are made, the pre-computed aggregations are already available, eliminating the need for time-consuming computations during query execution. This resolves the contradiction by preparing data in advance so that accurate aggregations can be retrieved instantly without latency.
Solution Approach 2:
The patent introduces an intermediary layer (aggregation cache or summary tables) between the raw data warehouse and the query interface. This intermediary pre-holds computed aggregations, allowing fast retrieval without directly querying the underlying data warehouse. The intermediary acts as a buffer that provides both accuracy (pre-computed values) and speed (instant retrieval), resolving the latency-accuracy tradeoff.
2Quantity of substance
If batch applications process large volumes of data, then complete data aggregation can be achieved, but processing speed and productivity decrease
Solution Approach 1:
The patent divides the large volume of data into smaller manageable segments or partitions. Instead of processing all data at once, the system processes segments independently and maintains aggregations for each segment. This segmentation allows parallel processing and reduces the time required to handle large data volumes while maintaining complete aggregation coverage across all segments.
Solution Approach 2:
The system computes and stores more aggregation data than immediately needed (pre-computing multiple levels of aggregation). By performing partial actions in advance (computing aggregations for different time periods, dimensions, and granularities), the system has pre-ready data available for various query types, eliminating the need to process all raw data whenever a query is made, thus improving productivity.
3Adaptability or versatility
If multiple applications individually retrieve data from data warehouse, then each application gets its required data, but system complexity and resource consumption increase
Solution Approach 1:
The patent merges multiple individual data retrieval operations into a single unified query interface. Instead of each application independently querying the data warehouse, the system provides a consolidated aggregation layer that serves multiple applications simultaneously. This combining approach reduces system complexity by centralizing data access and eliminating redundant retrieval operations while maintaining the ability to fulfill diverse application requirements.
Solution Approach 2:
The patent creates a universal aggregation service that can serve multiple different applications with varying data requirements through a single interface. The pre-computed aggregations are designed to be multi-functional, supporting various query types and applications without requiring separate processing paths. This universality reduces system complexity while maintaining adaptability to different application needs.
Data Source
AI summary
A method performed by a data processing system for processing data, the method including: intermittently receiving data from one or more data streams, the received data including data records; detecting two or more particular data records in the received data records, where the detected two or more particular data records each include a particular identifier; for that particular identifier, creating a collection of data records; for at least one particular data record included in the collection of data records, searching data records for a historical aggregation of data; and computing combined data; modifying a data record by inserting the combined data into a field of the data record and by inserting data from at least one of the data records in the collection into another field of the data record; based on applying the rules, writing to memory one or more instructions for initiation of one or more actions.