Search Query Sampling Analysis for Long Tail Data Distribution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing sampling analysis methods for search engine query data fail to accurately represent real distributions, especially for medium or small data sizes, due to 'long tail' characteristics, leading to underestimation of low-frequency query keywords and inaccurate user demand and market trend analysis.
Innovation Solution
A system and method that classifies query keywords into subsets based on page view values, calculates sample probabilities and sizes for each subset, and samples query data to ensure that low-frequency keywords are effectively represented, using a two-stage sampling process to closely resemble real data distributions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If existing sampling analysis method is used, then storage requirements are reduced, but the accuracy of representing real data distribution deteriorates, especially for low-frequency query keywords
Solution Approach 1:
The patent divides query keywords into different frequency groups (high-frequency, medium-frequency, and low-frequency groups) and applies different sampling rates to each group. This segmentation allows the system to use lower sampling rates for high-frequency keywords while using higher sampling rates for low-frequency keywords, thereby maintaining storage efficiency while improving the accuracy of representing the true data distribution, particularly for low-frequency keywords that would otherwise be underrepresented.
2Productivity
If existing sampling analysis method is used, then processing workload is reduced, but the reliability of user demand analysis deteriorates due to underestimation of low-frequency keywords
Solution Approach 1:
The patent changes the sampling rate parameter dynamically based on the frequency characteristics of different keyword groups. By adjusting the sampling rate from a fixed value to a variable that depends on keyword frequency, the system maintains low processing workload through sampling while ensuring that low-frequency keywords are sampled at higher rates, thereby improving the reliability of user demand analysis and preventing underestimation of important but less frequent search terms.
3Device complexity
If fixed sampling rate is used, then simplicity of implementation is maintained, but the quality of sampled data deteriorates for medium and small data sizes
Solution Approach 1:
The patent segments keywords into frequency-based groups and assigns different sampling rates to each segment. This approach maintains relative implementation simplicity by using clear frequency thresholds and fixed sampling rates for each group, while significantly improving sampled data quality for medium and small data sizes. The segmentation ensures that even with limited data, low-frequency keywords are adequately represented, providing a more accurate reflection of the true data distribution without requiring complex adaptive sampling algorithms.
Data Source
AI summary
Sampling analysis includes classifying a plurality of query keywords into a plurality of query keyword subsets according to page view (PV) values associated with the plurality of query keywords, the plurality of query keywords being submitted by a plurality of users; determining a respective plurality of sample rates of a respective plurality of query keywords in a respective one of the plurality of query keyword subsets; and sampling query data in the respective one of the plurality of query keyword subsets according to the respective plurality of sample rates.