Granularity-Adaptive Correlation Extraction in Databases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database systems are limited in extracting and processing correlation structures from data, as they primarily support scalar correlation coefficients, which are restricted to linear correlations under normal distribution assumptions, and incur high storage, access, and processing costs.
Innovation Solution
The method involves generating approximate empirical correlation representations (ACRs) at runtime, processing empirical correlation structures, and storing them in databases, allowing for the representation and handling of arbitrary correlation structures between univariate distributions using multi-dimensional histograms and copula-based techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If scalar correlation coefficients (Pearson's correlation, Spearman's Rho) are used to extract correlation from historic data, then linear correlations can be calculated, but the method is limited to normal distribution assumptions and cannot capture arbitrary correlation structures
Solution Approach 1:
The patent segments the continuous correlation structure into discrete bins forming a multi-dimensional histogram. Each bin represents a discrete correlation pattern, allowing the system to capture arbitrary correlation structures while maintaining computational efficiency through discrete representation rather than continuous functions.
Solution Approach 2:
The patent transforms the correlation representation from scalar parameters (traditional correlation coefficients) to a multi-dimensional histogram structure with multiple bins. This parameter transformation enables the representation of arbitrary correlation structures beyond linear correlations, including non-normal distribution patterns.
2Measurement precision
If detailed correlation structures are extracted and stored from historic data, then accurate correlation analysis can be performed, but storage, access, and processing costs increase significantly
Solution Approach 1:
The patent uses a finite number of bins (e.g., 5x5=25 bins for bivariate correlation) to represent the correlation structure, which is sufficient to capture essential correlation patterns without requiring exhaustive detail. This partial representation maintains accuracy while dramatically reducing storage and processing requirements compared to storing complete historical data or using complex continuous models.
3Loss of information
If complete historical data is stored to enable correlation extraction, then accurate correlation patterns can be derived, but storage costs and data access complexity increase
Solution Approach 1:
The patent extracts only the essential correlation structure from historical data and stores it as a compact multi-dimensional histogram, separating the correlation pattern from the raw historical data. This extraction approach preserves complete correlation information while reducing storage requirements from storing entire historical datasets to storing only the distilled correlation patterns.
Data Source
AI summary
Implementations include generating a first plurality of univariate distributions based on known data stored in a computer-readable database, each univariate distribution of the plurality of distributions comprising an arbitrary distribution, processing the univariate distributions and the known data to generate an empirical ACR provided as a multi-dimensional histogram, storing the empirical ACR in the computer-readable database, retrieving, from computer-readable memory, a second plurality of univariate distributions, each univariate distribution in the second plurality of univariate distributions being associated with a respective set of uncertain values that are to be correlated to a respective set of uncertain values associated with one or more other univariate distributions in the second plurality of univariate distributions, processing the empirical ACR and the second plurality of univariate distributions to generate a correlation histogram that represents a correlation between the respective sets of uncertain values and storing the correlation histogram in the computer-readable database.