Empirical Cumulative Distribution Function Estimator for Large Datasets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Constructing a stable histogram for large and continuously collected data samples, such as those from online surveys, is challenging due to unknown sample sizes, making it resource-consuming to represent the data effectively.
Innovation Solution
A method involving a programmed computer that iteratively adds random samples to calculate the empirical cumulative distribution function (ECDF) until the distance between ECDFs is below a threshold, determining a stable sample size for constructing a stable empirical distribution representation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all samples from a large continuously collected dataset are used to construct a histogram, then the accuracy of the empirical distribution representation is improved, but the computational resources and time required increase significantly
Solution Approach 1:
The patent applies partial action by calculating ECDFs on subsets of samples rather than the complete dataset. It iteratively processes portions of the data, computing ECDFs on progressively larger subsets until convergence is achieved, thus avoiding the need to process all samples while still obtaining an accurate empirical distribution representation.
Solution Approach 2:
The patent performs preliminary action by pre-calculating ECDFs on sample subsets before final analysis. It computes ECDFs on initial sample sets, compares them to assess stability, and uses these preliminary results to determine whether additional processing is needed, thereby reducing overall computational requirements.
2Measurement precision
If all samples from a large continuously collected dataset are used to construct a histogram, then the accuracy of the empirical distribution representation is improved, but the computational resources required increase significantly
Solution Approach 1:
The patent applies partial action by calculating ECDFs on subsets of samples rather than the complete dataset. It iteratively processes portions of the data, computing ECDFs on progressively larger subsets until convergence is achieved, thus avoiding the need to process all samples while still obtaining an accurate empirical distribution representation.
3Loss of time
If a small number of samples is used to construct a histogram, then the computational resources and time are reduced, but the stability of the empirical distribution representation deteriorates
Solution Approach 1:
The patent applies feedback by iteratively computing ECDFs on progressively larger sample subsets and comparing consecutive results. It uses the difference between successive ECDFs as feedback to determine whether the empirical distribution has converged to a stable representation, automatically stopping when sufficient stability is achieved without requiring excessive samples.
Solution Approach 2:
The patent applies dynamics by making the sample size adaptive rather than fixed. It starts with a small sample set and dynamically increases the number of samples processed based on the stability assessment of the empirical distribution, allowing the system to adaptively find the minimum sufficient sample size for accurate representation.
4Productivity
If the sample size is reduced for processing large datasets, then the computational efficiency is improved, but the accuracy of representing the original distribution may be compromised
Solution Approach 1:
The patent applies feedback by iteratively computing ECDFs on progressively larger sample subsets and comparing consecutive results. It uses the difference between successive ECDFs as feedback to determine whether the empirical distribution has converged to a stable representation, automatically stopping when sufficient stability is achieved without requiring excessive samples.
Data Source
AI summary
A method includes obtaining via a programmed computer, a first set of n random samples and a second set of n+k random samples from a base set of samples where k is a lag, iteratively adding more random samples to the first and second sets from the base set via the programmed computer, obtaining a distance between the first and second sets of random samples by calculating via the programmed computer, an empirical cumulative distribution function (ECDF) for the first and second sets in each iteration until the distance between the ECDFs is below a threshold, and constructing a stable empirical distribution representation via the programmed computer using a number of samples that is a function of the first and second sets whose distance is below the threshold.