Back-Calculation of Aggregates Using Historical Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data processing systems face challenges in calculating real-time aggregates efficiently, especially when insufficient data is available, as they often require extensive time and resources to collect and process data, leading to delays in making data available.
Innovation Solution
The method involves back-calculating near real-time aggregates by retrieving historical data from a storage system, identifying relevant events, and updating the aggregate values based on detected events, using a batch processing module to reuse previously collected data, thereby reducing the need for repeated data retrieval and conserving computational resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If data processing systems collect and process data in real-time for new aggregates, then measurement precision is improved, but loss of time increases due to the extensive collection period required
Solution Approach 1:
The system performs preliminary actions by collecting and storing historical event data in advance in data stores. When a new aggregate is created, the system can immediately query this pre-collected historical data to calculate aggregate values without waiting for real-time data accumulation, thus resolving the contradiction between measurement precision and time loss.
2Measurement precision
If data processing systems retrieve historical data repeatedly for each aggregate calculation, then measurement precision is improved, but use of energy increases due to repeated data retrieval operations
Solution Approach 1:
The system prepares and stores historical event data in advance in data stores during normal operations. When aggregate calculations are needed, the system queries this pre-prepared data rather than retrieving it repeatedly from source systems, significantly reducing energy consumption while maintaining calculation accuracy.
Solution Approach 2:
The system creates copies of historical event data and stores them in data stores for efficient retrieval. Instead of repeatedly accessing original data sources, the system queries these copied datasets, reducing the computational energy required for each aggregate calculation while preserving data accuracy.
3Manufacturing precision
If data processing systems wait for sufficient data to be collected before calculating aggregates, then manufacturing precision is improved, but productivity decreases due to delays in making data available
Solution Approach 1:
The system performs preliminary data collection and storage actions by maintaining historical event data in data stores during normal system operations. When a new aggregate is created, the system can immediately query this pre-collected data to calculate accurate aggregate values without waiting for the traditional data collection period, thus improving productivity while maintaining manufacturing precision.
Data Source
AI summary
Described are techniques for back-calculating one or more values of a new, real-time aggregate before sufficient data to calculate the new, real-time aggregate has been collected, wherein the back-calculating is based on data collected for one or more aggregates that have been executing prior to start of execution of the new, real-time aggregate.