In-Memory Counter Mapping for Data Stream Monitoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing network monitoring systems face inefficiencies in processing and analyzing large volumes of financial transaction data from message queues, particularly in calculating transaction rates and counters, which leads to high system overhead and resource utilization due to frequent database queries.
Innovation Solution
A method and apparatus that utilize an in-memory data structure with counters to map and update values from a data stream, allowing for real-time monitoring and reporting of transaction rates by copying counter values to a user-accessible data structure at fixed intervals, reducing the need for frequent database queries and minimizing system overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If frequent database queries are used to calculate transaction rates and counters, then measurement precision of transaction monitoring is improved, but system overhead and resource utilization increase
Solution Approach 1:
The system segments the monitoring function into two parts: a high-speed in-memory counter that captures individual transactions, and a periodic aggregation mechanism that calculates rates. This segmentation allows precise transaction counting without requiring frequent database queries, thus reducing system overhead while maintaining measurement precision.
Solution Approach 2:
Instead of continuously querying the database for transaction rates, the system uses periodic action by aggregating counter values at fixed time intervals. This approach maintains accurate transaction monitoring while significantly reducing the frequency of database interactions and lowering system overhead.
2Reliability
If frequent database queries are performed to obtain transaction data, then reliability of transaction rate calculation is improved, but resource utilization increases
Solution Approach 1:
The system performs preliminary action by maintaining in-memory counters that continuously capture transaction data as it occurs. This preliminary counting eliminates the need for repeated database queries to gather transaction data, ensuring reliable transaction rate calculation while minimizing resource utilization during the monitoring process.
Solution Approach 2:
The system creates a copy of transaction data in the form of in-memory counter structures that mirror the essential counting functionality. This copying approach allows reliable transaction rate calculation by working with lightweight in-memory representations rather than repeatedly accessing the heavy database, thus reducing resource utilization.
3Speed
If real-time monitoring is implemented with frequent data access, then speed of monitoring is improved, but system overhead increases
Solution Approach 1:
The system extracts the essential counting function from the database and places it in lightweight in-memory counter structures. This extraction enables high-speed real-time monitoring by operating on fast in-memory data while eliminating the overhead of frequent database access, thus achieving speed improvement without proportionally increasing system overhead.
Data Source
AI summary
Systems, methods, and apparatus for monitoring flow of data traffic are described. For each of a plurality of transfer units from a data stream (e.g., messages from a message queue), a value of a first field of the transfer unit is mapped to a corresponding counter in random-access memory and the counter is updated. Counter values are periodically copied to a user-accessible region. In one example, message field values are obtained from database log files. In another example, message field values are obtained via an interface called by a database management system.