Data Collector Memory Management via Embedded Database
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data collector jobs in performance analytics software applications face challenges due to increasing database sizes and the number of key performance indicators (KPIs), leading to excessively high main memory and processing resource utilization, resulting in prolonged execution times and potential job failures.
Innovation Solution
The solution involves splitting data collector jobs into stages and storing intermediate results in an embedded database when main memory utilization reaches a threshold, allowing for the release of main memory and enabling the next stage of the job to proceed, until all relevant records are processed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data collector jobs process all records in main memory, then complete KPI calculations can be achieved, but main memory utilization becomes excessively large
Solution Approach 1:
The data collector job is divided into multiple stages: reading records in chunks, performing partial aggregations, storing intermediate results in an embedded database, and finally computing complete KPIs. This segmentation allows processing large datasets without loading everything into main memory simultaneously.
Solution Approach 2:
The solution introduces a new storage dimension by using an embedded database (H2 database) to store intermediate aggregation results. This transitions the system from a single-memory-dimension approach to a multi-dimensional storage architecture, enabling persistent storage of intermediate states without consuming excessive main memory.
2Quantity of substance
If data collector jobs use large main memory, then all records can be processed, but execution time becomes excessively long
Solution Approach 1:
By segmenting the processing into stages with intermediate storage, the system avoids the overhead of processing entire datasets in single passes. Each stage processes manageable chunks and persists results, enabling efficient resource utilization and faster overall execution.
Solution Approach 2:
The embedded database allows the data collector to persist intermediate results and resume processing without losing progress. This continuity enables the system to maintain useful action across multiple execution phases without redundant reprocessing, reducing total execution time.
3Reliability
If data collector jobs consume excessive main memory, then complete KPI calculations can be performed, but other applications are starved of resources
Solution Approach 1:
The processing job is segmented into stages that use limited main memory at any given time, with intermediate results stored in the embedded database. This ensures that other applications can access sufficient memory resources while the KPI calculations are completed through multiple staged operations.
Solution Approach 2:
The embedded database acts as an intermediary storage layer between the data collector and main memory. It enables the system to maintain complete KPI calculation capabilities while decoupling the memory requirements from the processing logic, allowing other applications to function normally.
4Quantity of substance
If data collector jobs process large datasets, then comprehensive KPIs can be generated, but virtual memory thrashing occurs
Solution Approach 1:
The large dataset processing is segmented into manageable chunks that fit within available memory. Intermediate aggregation results are persisted in the embedded database, preventing the need to keep all data in memory simultaneously and eliminating virtual memory thrashing.
Solution Approach 2:
The system performs preliminary aggregations on data chunks before storing them in the embedded database. This preliminary processing reduces the volume of data that needs to be handled in subsequent stages, preventing memory thrashing during final KPI computation.
Data Source
AI summary
An example embodiment may involve a main database; a main memory; and one or more processors configured to: retrieve, by a data collector application, records from the main database, wherein the data collector application includes an embedded database; aggregate, by the data collector application, values in the records relating to a key performance indicator (KPI) to form partial KPI data stored in one or more blocks of the main memory; determine, by the data collector application, that utilization of the main memory exceeds a pre-defined threshold; in response to the utilization of the main memory exceeding the pre-defined threshold, write, by the data collector application, the partial KPI data to a row of the embedded database; and release, by the data collector application, the one or more blocks of the main memory used to store the partial KPI data.