Polynomial Data Batching for Query Speed
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for querying large datasets are time-consuming and computationally intensive, requiring days to retrieve and process data, especially as data volumes grow, leading to significant delays in decision-making and data representation.
Innovation Solution
The method involves batching data based on fluctuation, fitting each batch with a polynomial model, and generating response data using these models to process queries efficiently, eliminating the need for database retrieval by storing data as aggregated polynomials indexed by ranges.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional database querying methods are used to retrieve large datasets, then data accuracy and completeness are maintained, but query time and computational resources increase significantly
Solution Approach 1:
The patent applies preliminary action by pre-processing the dataset before storage, dividing it into batches and fitting polynomial models to each batch in advance. This preparation work is performed once during data ingestion, allowing subsequent queries to execute rapidly by evaluating the pre-computed models rather than retrieving and processing raw data at query time.
Solution Approach 2:
The patent creates simplified copies of the original data in the form of polynomial models that approximate the underlying patterns. These model copies retain the essential information needed for querying while being much more efficient to evaluate, trading some detailed precision for significant speed improvements in data retrieval and analysis.
2Loss of information
If the dataset size increases to capture more information, then data completeness improves, but the time and compute resources required for querying increase
Solution Approach 1:
The patent segments the large dataset into multiple smaller batches based on data characteristics such as time ranges or value ranges. Each batch is independently processed and fitted with its own polynomial model. This segmentation allows the system to handle large datasets by breaking them into manageable pieces, improving both storage efficiency and query performance through localized model evaluation.
Solution Approach 2:
The patent transforms the raw data parameters into polynomial model parameters through curve fitting. By changing the representation from raw data points to polynomial coefficients, the system achieves a compact form that captures the essential patterns while enabling rapid evaluation. The polynomial degree and batch size parameters can be adjusted to balance accuracy and performance.
3Speed
If polynomial model fitting is applied to batch data, then query speed improves significantly, but model complexity and initial processing requirements increase
Solution Approach 1:
The patent applies partial action by using polynomial models of limited degree (e.g., quadratic or cubic) rather than attempting to capture all possible data variations. This selective approximation achieves sufficient query speed improvement while avoiding the excessive complexity of higher-degree polynomials or more sophisticated models, finding the optimal balance between speed and accuracy for the specific use case.
Data Source
AI summary
Methods and systems for querying data are disclosed. In disclosed embodiments, a data set is batched to a first data set in a first range and a second data set in a second range, the batching based on a determination of data fluctuation of the data set. A first model that includes an m order polynomial based on the first data set and a second model that includes an n order polynomial based on the second data set is generated. In embodiments, a query of the data set is received, spanning at least a portion of the range of the first and second data sets. A query output is generated in response to the query, based on output from the first model and second model.