Multi-Tier Web Analytics System for Low Latency Reporting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current web site analytics systems suffer from high latency, making it difficult to monitor sudden traffic spikes or drops and track changes in web site traffic data in a timely manner, which hinders publishers' ability to view web traffic data and detect errors in analytics deployment.
Innovation Solution
A multi-tiered real-time analytics system that combines low latency web site traffic data with historical data, using a collecting tier, logging tier, persistent tier, real-time analytics tier, and front-end tier to process and report web site traffic data with low latency, allowing for real-time reporting and segmentation of data over a rolling window of time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing web site analytics data systems process high volume of web traffic data with high fidelity, then reporting accuracy is improved, but latency increases
Solution Approach 1:
The system segments web traffic data processing into multiple tiers: collecting tier for data reception, logging tier for persistent storage, real-time analytics tier for low-latency processing, and conventional analytics tier for high-fidelity analysis. This segmentation allows different processing paths for different requirements, resolving the contradiction between speed and accuracy.
Solution Approach 2:
The system introduces intermediary components including a data warehouse as buffer between data collection and analysis, and a query service that mediates between user requests and analytics processing. These intermediaries decouple the timing between data generation and reporting, reducing latency while maintaining accuracy.
2Quantity of substance
If existing analytics systems report data over substantial periods of time, then data completeness is improved, but ability to monitor sudden traffic changes deteriorates
Solution Approach 1:
The system implements dynamic time window processing where the real-time analytics tier processes data over rolling time windows that can be adjusted based on traffic patterns. This allows the system to adapt between monitoring short-term sudden changes and long-term trends, resolving the contradiction between data completeness and response speed.
3Productivity
If publishers use existing analytics systems, then comprehensive traffic analysis is achieved, but ability to detect errors and changes timely deteriorates
Solution Approach 1:
The system implements feedback mechanisms where the real-time analytics tier continuously monitors traffic patterns and immediately alerts publishers to sudden changes or anomalies. This real-time feedback loop enables timely error detection while the conventional analytics tier provides comprehensive historical analysis, resolving the contradiction between analysis depth and detection speed.
Data Source
AI summary
Machine-implemented methods for providing information to a graphical user interface for displaying web site traffic information are provided. In one aspect, a method includes acquiring low latency web site traffic data for a web site and acquiring historical web site traffic data for the web site. The method also includes combining, into combined web site traffic data, the low latency web site traffic data for the web site with the historical web site traffic data for the web site, and providing the combined web site traffic data to a graphical user interface for display. Systems, graphical user interfaces, and machine-readable media are also provided.