Website Statistics Measurement Data Bucket Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current website statistics solutions require high-capacity systems for processing and storing large volumes of measurement data, leading to performance bottlenecks and high memory usage due to sequential processing.
Innovation Solution
A system and method for processing website statistics that involves partitioning measurement data into visitor buckets for parallel processing, using multiple processors or cores, and optimizing storage and reporting by distributing data to bucket writers for efficient processing and storage, reducing memory usage and processing time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If full measurement data is read in memory for processing using a single processor, then processing can be completed, but memory usage becomes excessively high and processing time increases
Solution Approach 1:
The patent divides measurement data into multiple buckets that are processed separately. Each bucket contains a subset of the total measurement data, allowing the system to process one bucket at a time in memory rather than loading all data simultaneously. This segmentation enables complete data processing while keeping memory usage at manageable levels.
Solution Approach 2:
The patent introduces a new dimension of processing by organizing data into bucket structures with multiple levels (buckets containing sub-buckets containing measurements). This hierarchical organization allows the system to navigate through data in a structured manner, processing only the necessary portion in memory at any given time while maintaining access to the complete dataset.
2Measurement precision
If full measurement data is read in memory for processing using a single processor, then processing can be completed, but processing time becomes excessively long
Solution Approach 1:
The patent segments measurement data into multiple buckets that can be processed in parallel or sequentially without loading the entire dataset into memory at once. This segmentation enables faster processing by avoiding the bottleneck of loading and processing all data through a single processor sequentially, while still ensuring complete data processing through systematic bucket iteration.
Solution Approach 2:
The patent performs preliminary organization of measurement data into bucket structures before processing. This preliminary action groups related measurements together in advance, so that during the actual processing phase, the system can efficiently access and process pre-organized data subsets without the overhead of organizing data during processing, thereby reducing total processing time.
3Productivity
If high capacity systems with more CPU power and memory are used, then processing capability improves, but system cost and complexity increase
Solution Approach 1:
The patent employs segmentation of measurement data into buckets, which enables efficient processing using modest system resources. By processing one bucket at a time rather than requiring all data to reside in memory simultaneously, the system achieves effective processing capability without needing proportionally larger CPU power or memory capacity, thus avoiding increased system complexity and cost.
Solution Approach 2:
The patent uses bucket structures that can be efficiently copied and processed. Instead of requiring large amounts of physical memory to hold all measurement data, the system loads smaller bucket copies into memory, processes them, and discards them. This copying approach maintains processing capability while avoiding the need for high-capacity memory systems.
Data Source
AI summary
A method and a system for processing measurement data for website statistics are provided. The measurement data is processed in parallel bucket writers and stored in buckets. Upon receiving a report request the buckets are processed in parallel bucket queriers to obtain report data.