In-Database Vectorized UDTFs for Distributed Correlation Calculation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional database systems face challenges in efficiently accessing and processing vector spaces for mathematical calculations, particularly in generating Pearson correlation matrices, due to limitations in row-based or column-based access methods, which hinder the effective utilization of distributed computing resources.
Innovation Solution
The system employs user-defined table functions (UDTFs) to decompose summations in the Pearson product-moment formula into linear algebra-based dot products, enabling vector chaining and distributed computing resources, thereby facilitating efficient vector processing and correlation calculations across a network-based database system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional row-based or column-based access methods are used in database systems, then data storage and retrieval are maintained with simple structures, but the ability to efficiently access and process vector spaces for mathematical calculations is limited
Solution Approach 1:
The patent segments vector data into separate vector tables with dedicated access paths, decoupling vector operations from traditional row-based access. Each vector is stored in its own table with direct indexing capabilities, allowing independent access to vector components without traversing complete rows or columns.
Solution Approach 2:
The patent introduces a new dimensional approach by storing vectors in specialized vector tables with their own indexing dimensions, separate from the traditional row-column grid. This enables direct access to vector spaces through a new dimensional layer that supports mathematical operations efficiently.
2Productivity
If distributed computing resources are utilized for correlation calculations, then computational speed and processing capability are improved, but the complexity of implementing and managing the distributed system increases
Solution Approach 1:
The patent divides the correlation calculation task into segments that can be distributed across multiple computing nodes. Each node processes a portion of the vector operations independently, and results are aggregated to produce the final correlation matrix, enabling parallel processing without requiring complex centralized coordination.
Solution Approach 2:
The patent introduces vector tables as intermediary structures that mediate between data storage and computational operations. These tables provide a standardized interface for vector access and manipulation, simplifying the implementation of distributed computing by providing a common data representation layer.
3Quantity of substance
If vector operations are performed on large datasets, then the correlation matrix calculation becomes more comprehensive, but execution time and computational complexity increase significantly
Solution Approach 1:
The patent segments large datasets into vector tables that can be processed in parallel across multiple nodes. Each vector table can be independently accessed and processed, allowing the system to handle large datasets by distributing the workload rather than processing everything sequentially on a single node.
Solution Approach 2:
The patent performs preliminary organization of data into vector tables with optimized access paths before performing correlation calculations. This pre-processing step structures the data in a way that enables faster retrieval and computation, reducing the time required for actual correlation analysis on large datasets.
Data Source
AI summary
The subject technology determines a set of shards of rows from a data set based on a number of rows and a number of execution nodes to execute a request for determining a correlation. For each shard from the set of shards, the subject technology sends a particular user defined table function (UDTF), including a particular shard of rows, to a different execution node to perform a set of operations for determining the correlation. The subject technology provides a set of output values of each particular UDTF corresponding to each shard from the set of shards in a second UDTF. The subject technology sends the second UDTF to a particular execution node to perform an aggregate operation using the set of output values of each particular UDTF. The subject technology receives a value of the correlation from the particular execution node based on the aggregate operation.