Multivariate Testing Data Aggregation via Web Crawling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current multivariate testing systems lack the ability to gather and analyze data across multiple websites and testing services, limiting the evaluation of website changes to individual sites and preventing comprehensive analysis of user behavior and conversion rates.
Innovation Solution
A method and system for tracking and gathering multivariate testing data by crawling through servers, analyzing page views, and saving data related to multivariate tests in a data store, allowing for the collection and analysis of testing data from multiple webpages and services, including A/B and split tests, across different websites.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multivariate testing is performed on individual websites using separate testing services, then each service can independently test and analyze its own data, but the ability to gather and analyze testing data across multiple websites is limited
Solution Approach 1:
The system creates a universal data collection framework that can gather multivariate testing data from multiple different testing services and websites through a single platform. The web crawler and data extraction module are designed to work with various testing services uniformly, enabling cross-site data aggregation while maintaining the independence of each testing service
Solution Approach 2:
The patent introduces an intermediary system (the data gathering and analysis platform) that acts as a mediator between multiple testing services and the end user. This intermediary collects data from various sources, standardizes the data format, and provides unified analysis capabilities without requiring changes to the individual testing services
2Adaptability or versatility
If data is collected from multiple websites and testing services, then comprehensive analysis capability is improved, but system complexity increases
Solution Approach 1:
The system divides the complex data collection process into distinct modular components: a web crawler module for navigating websites, a data extraction module for gathering testing data, a data storage module for saving collected data, and an analysis module for processing the data. Each module handles a specific task independently, reducing overall system complexity while enabling comprehensive data collection
3Loss of information
If testing data is aggregated from multiple sources, then evaluation of website changes across sites is improved, but data processing time increases
Solution Approach 1:
The system performs preliminary data collection and storage operations continuously, maintaining a ready pool of testing data from multiple sources. When analysis is requested, the pre-collected data can be quickly processed without requiring time-consuming data gathering operations, thus reducing analysis latency while maintaining comprehensive data coverage
Data Source
AI summary
A system and method for tracking and gathering data respective of multivariate testing on a plurality of webpages are provided. The method includes crawling through a plurality of servers hosting the plurality of webpages; for each uniform resource locator (URL) of a webpage of the plurality of webpages encountered during the crawling: sending a request to download the webpage identified by the URL; downloading at least one page view of the webpage; analyzing the at least one downloaded page view to identify data related to at least a multivariate test; and saving data identifying the at least a multivariate test performed in the plurality of webpages in a data store.