Pick-and-Drop Sampling for Data Stream Frequency Moments
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing algorithms require polynomial space to approximate frequency moments for k>2, limiting their efficiency in data stream processing.
Innovation Solution
A method using non-uniform sampling, specifically 'pick-and-drop sampling,' to approximate frequency moments in a data stream, reducing space complexity to O(n1−2/k log(n)) bits by arranging data into a matrix and selecting elements based on presence and frequency, allowing for a single pass through the data stream.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If polynomial space is used to approximate frequency moments for k>2, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The algorithm segments the data stream processing into distinct phases: sampling phase where elements are selected based on frequency thresholds, and counting phase where frequency moments are computed from the sampled subset. This segmentation allows approximate frequency moment computation with reduced space requirements compared to processing the entire stream with polynomial space
Solution Approach 2:
The algorithm changes the parameter of space complexity from polynomial O(n^ε) to sublinear O(n^(1-2/k) log n) by modifying the sampling strategy. Instead of uniform sampling or processing all elements, it uses frequency-based sampling where the sampling probability and threshold parameters are adjusted based on the moment order k, achieving the desired space reduction while maintaining approximation guarantees
2Measurement precision
If polynomial space is used to approximate frequency moments for k>2, then measurement precision is improved, but productivity decreases
Solution Approach 1:
The algorithm extracts a representative subset of elements from the data stream based on frequency sampling criteria, processing only this extracted subset to compute frequency moments. This extraction principle avoids processing the entire polynomial-sized stream, thereby improving processing efficiency and productivity while maintaining measurement precision through careful sampling design
Solution Approach 2:
The algorithm performs partial action by computing frequency moments from a sampled subset rather than the complete data stream. The sampling rate and subset size are tuned to provide sufficient statistical accuracy for frequency moment estimation, achieving the required measurement precision with less computational work and improved productivity
Data Source
AI summary
A database system includes an input to a database server configured to deliver a data stream formed of a sequence of elements, D={p1, p2, . . . , pm} of size m of numbers from {1, . . . , n} to the database server. The system further includes a computer program that causes a processor to approximate frequency moments (Fk) in the data stream, such that a frequency of an element (i) is defined as fi=|{j:pj=i}| and a k-th frequency moment of D is defined asFk=∑i=1nmikinasingle pass through the data stream. The processor is caused to carry out the steps of locating elements (i) with a frequency ΩFk in the data stream as heavy elements and approximating fi as ≧ a fraction of fi to limit memory resources used by the processor to estimate Fk to O(n1−2/k log(n)) bits.