Lossy Hit Buffer for Real-Time Web Analytics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current web site analytics systems face high latency in reporting, making it difficult to monitor sudden traffic spikes or drops and track link propagation, and often result in delayed detection of errors in analytics tracking code deployment.
Innovation Solution
A multi-tiered real-time analytics system employing a configurable buffer with a lossy storage scheme to store and transmit event tracking data communications, allowing for batch transmission and discarding of overflow requests to minimize latency and optimize memory, latency, and network bandwidth.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a high volume of web traffic data is processed with high fidelity, then measurement precision is improved, but latency increases
Solution Approach 1:
The system segments the data processing into two paths: a lossy buffer for real-time low-latency processing and a persistent store for high-fidelity long-term storage. This allows the system to provide immediate feedback through the lossy buffer while maintaining accurate historical records in the persistent store, resolving the contradiction between speed and precision.
Solution Approach 2:
The system changes the data retention parameter from 100% fidelity to acceptable lossiness in the real-time buffer, allowing faster processing. The lossy buffer discards duplicate or redundant data, accepting some measurement imprecision in exchange for significantly reduced latency and improved throughput.
2Speed
If real-time processing is implemented to reduce latency, then speed is improved, but device complexity increases
Solution Approach 1:
The lossy buffer acts as an intermediary component between the data source and the persistent store. It receives data, performs real-time processing, and forwards to the persistent store only when necessary. This intermediary layer simplifies the overall system architecture by providing a single point for real-time processing logic, avoiding the complexity of continuous processing at every layer.
3Productivity
If a buffer is used to store event tracking data communications, then productivity is improved, but loss of information occurs when the buffer is full
Solution Approach 1:
The lossy buffer deliberately discards duplicate or redundant data when full, using a token bucket algorithm to manage capacity. This controlled discarding maintains high throughput by preventing buffer overflow, while the persistent store recovers and preserves the essential data history. The system accepts some information loss in the buffer to maintain overall productivity.
4Quantity of substance
If batch transmission is used to reduce network congestion, then network bandwidth efficiency is improved, but latency increases
Solution Approach 1:
The system uses periodic action by transmitting data in batches at predetermined intervals or when the lossy buffer fills up. This periodic batch transmission consolidates multiple smaller network requests into fewer larger requests, improving network bandwidth utilization while keeping latency acceptable through timely flushing of the lossy buffer.
Data Source
AI summary
Configurations providing a configurable buffer for storing incoming event tracking data communications in a lossy manner are described. In one aspect, a server can utilize the configurable buffer for storing the incoming event tracking data communications. When the buffer becomes full, the server can transmit the accumulated tracking data communications in the configuration buffer as a batch transmission. The server can discard any new incoming requests once the buffer becomes full. Further, the server can replace data in the buffer with new incoming requests based on one or more criterion if the buffer is full.In some implementations, the server transmits the batch of the accumulated tracking data communications in the configurable buffer after a predetermined time has elapsed.