Analytics Data Aggregation via Context Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Web analytics data processing lags behind due to the increasing size of logfiles from high web traffic, making it difficult to process data in real time, especially for sites with heavy traffic.
Innovation Solution
A distributed computing and data storage system that includes web servers, analytics systems with log processors, aggregation servers, and data servers to process and aggregate web analytics data, allowing for incremental updates of context-insensitive and context-sensitive analytics data, with context-sensitive updates occurring at specific times like the end of a day, and utilizing customizable aggregate tables built on the fly.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If logfiles are constantly growing due to high web traffic, then the quantity of data increases, but the processing speed decreases
Solution Approach 1:
The system divides the analytics data processing into multiple segments by separating context-insensitive data (processed continuously in real-time) from context-sensitive data (processed periodically at end-of-day). This segmentation allows different processing speeds for different data types, resolving the contradiction between handling large data volumes and maintaining processing speed.
Solution Approach 2:
The system performs preliminary processing of context-insensitive analytics data continuously as it arrives, rather than waiting for the entire logfile to be processed. This preliminary action reduces the processing burden on subsequent batch processing and enables faster overall data availability.
2Loss of time
If analytics data is processed continuously, then the freshness of data improves, but the processing time required increases
Solution Approach 1:
The system segments processing operations into two distinct streams: continuous processing for context-insensitive data (prioritizing freshness) and periodic batch processing for context-sensitive data (accepting longer processing time). This segmentation resolves the contradiction by not requiring all data to be processed immediately.
Solution Approach 2:
The system employs periodic batch processing at the end of each day to process context-sensitive analytics data. This periodic action allows the system to maintain data freshness for critical real-time metrics while dedicating specific time periods to comprehensive data processing, thereby reducing overall processing time requirements.
3Productivity
If the system processes all analytics data in real-time, then the speed of data availability improves, but the system complexity increases
Solution Approach 1:
The system segments analytics data into context-insensitive and context-sensitive categories, applying different processing strategies to each. This segmentation simplifies the overall system architecture by allowing specialized processing paths for different data types, reducing the complexity that would otherwise arise from attempting to process all data uniformly in real-time.
Solution Approach 2:
The system applies different processing qualities to different data streams: high-speed continuous processing for context-insensitive data and periodic batch processing for context-sensitive data. This local quality approach optimizes performance for each data type while managing system complexity through targeted processing rather than uniform treatment.
Data Source
AI summary
A method and system for aggregating analytics data is discussed. The system differentiates between analytics data that is context sensitive and therefore cannot be reliably updated incrementally (e.g., unique page views, time on site, etc.) and analytics data that is not context sensitive. The system aggregates the context insensitive metrics and dimensions incrementally, while aggregating the context sensitive metrics and dimensions after a specified time duration, such as at the end of the day. It is estimated that less than 10% of all metrics and dimensions are context sensitive. In this way, the aggregator server 160 is able to improve the freshness of more than 90% of the analytics data (represented by the context insensitive metrics and dimensions) to a shorter period of time than the prior art. Further, this reduces the possibility of over-counting metrics.