Big Data Analytics via Stratified Sampling and Feature Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current analytics platforms face challenges in efficiently processing large and streaming big data due to resource constraints, making it impractical to perform complex computations, leading to defective data if not addressed.
Innovation Solution
The system employs Approximate Query Processing (AQP) combined with incremental and non-incremental learning algorithms, stratified sampling, and feature selection to generate representative samples from big data, enabling efficient and accurate analytics without processing the entire dataset.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If complex computations are performed on all big data points, then analytical accuracy is improved, but computational resources and time requirements become prohibitive
Solution Approach 1:
The system extracts a representative subset of data points from the complete big data set using stratified sampling. This extracted subset is then used for computational analysis, thereby reducing the quantity of data that must be processed while maintaining analytical accuracy through careful selection of representative samples that reflect the overall data distribution.
Solution Approach 2:
The system performs computations on only a partial subset of the complete data set rather than all data points. By using stratified sampling to select a representative portion of the data, the system achieves sufficient analytical accuracy with reduced computational resources, applying partial action that is adequate for the analytical purpose without the excess of processing every single data point.
2Productivity
If data is moved from storage location to computing resource, then processing capability is improved, but time and resource constraints make it impractical for big data sets
Solution Approach 1:
Instead of moving the entire big data set from storage to computing resources, the system extracts a representative subset of data points using stratified sampling. This extracted subset is then transferred and processed, dramatically reducing data transfer time while maintaining processing capability through careful selection of representative samples that preserve the statistical properties of the complete data set.
3Productivity
If traditional computational platforms are used for big data analysis, then data processing can be performed, but the platforms lack the capacity to handle extremely large data sets efficiently
Solution Approach 1:
The system extracts a representative subset of data from the complete big data set using stratified sampling based on predictor variables. This extracted subset can then be processed using traditional computational platforms without requiring enhanced platform capacity, thereby improving data processing efficiency while avoiding the need for complex high-capacity infrastructure.
Solution Approach 2:
The system changes the parameter of data volume by selecting a representative subset rather than processing the complete data set. By using stratified sampling to reduce the data volume parameter while maintaining representativeness, traditional computational platforms can efficiently process the reduced data set without requiring enhanced capacity or complex infrastructure.
Data Source
AI summary
An apparatus for estimating analytics and interactive exploration of big data, stored and/or streaming, using approximate query processing is presented. The apparatus comprises a model constructor and a sampler. The model constructor identifies important predictors variables in big data using feature selection, predictor variables, and outcome variables and partitions the important predictor variables into one or more stratifications based either the identified interactions or identified relationships. The sampler generates a subset of data by querying the big data using a query constructed based on at least one stratification. The subset of data can be fed into an analytics generator. The analytics generator generates analytics data for the outcome variables based on the subset of data and an analytics algorithm and a visualization, e.g. an interactive visualization, comprising the outcome variables, the important predictor variables, the stratification, the subset of data, and the analytics data.