SQL Vector Queries Using Micro-Partition Pruning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional database management systems require significant resources and infrastructure investments, are susceptible to data loss during disasters, and have limited scalability, making them costly and inefficient for data storage and access.
Innovation Solution
A network-based database system utilizing compute service managers, execution platforms, and metadata organization to manage and execute queries efficiently, enabling granular selection and pruning of micro-partitions to reduce computing resources and query response time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional relational database management systems are used, then data storage and access are enabled, but significant computing and storage resources are required with limited scalability
Solution Approach 1:
The database is divided into micro-partitions that are independently manageable and can be selectively processed. This segmentation allows the system to handle large datasets by processing only relevant micro-partitions rather than the entire database, thereby improving scalability while reducing computing resource consumption.
Solution Approach 2:
The patent introduces a vector data type with dimensional components (embedding dimension, feature dimension) to represent data in a higher-dimensional space. This dimensional transformation enables more efficient data organization and query processing, allowing the system to scale effectively while optimizing resource usage through vectorized operations.
2Quantity of substance
If traditional database infrastructure is deployed, then data storage is achieved, but significant capital investment in hardware and infrastructure is required
Solution Approach 1:
The database system is designed to support multiple data types including traditional relational data and new vector data types within a unified architecture. This universality allows the same infrastructure to handle diverse data workloads without requiring separate specialized systems, thereby reducing overall infrastructure investment while maintaining extensive data storage capacity.
Solution Approach 2:
The patent employs virtualization and cloud-based deployment models where physical infrastructure is replicated virtually across multiple nodes. This copying approach allows the system to achieve high storage capacity through virtual instances rather than requiring proportional physical hardware investment, reducing capital expenditure while maintaining scalability.
3Quantity of substance
If traditional database systems are used, then data is stored and accessed, but the database is highly susceptible to data loss during power outages or disasters
Solution Approach 1:
The system implements automated backup mechanisms that create copies of data and micro-partitions before failures occur. These pre-positioned backups act as a cushion against data loss during power outages or disasters, ensuring that data can be restored without significant loss even when primary storage becomes unavailable.
Solution Approach 2:
The patent introduces a cloud-based intermediary layer that acts as a buffer between the primary database and potential failure points. This intermediary infrastructure includes distributed backup systems and fault tolerance mechanisms that mediate against direct exposure to hardware failures, power outages, and disasters, thereby improving data reliability while maintaining storage capacity.
4Loss of information
If queries are executed against large datasets, then data extraction is achieved, but query response time increases without efficient pruning mechanisms
Solution Approach 1:
The system extracts and processes only the necessary micro-partitions containing relevant data for each query rather than scanning the entire database. This selective extraction approach maintains complete data extraction for the query scope while dramatically reducing query response time by avoiding unnecessary processing of irrelevant data portions.
Solution Approach 2:
The patent implements partial processing by identifying and executing only the necessary micro-partition operations required to answer a query. This partial action strategy avoids the excessive action of processing entire datasets, thereby reducing query response time while maintaining sufficient data extraction completeness for accurate query results.
Data Source
AI summary
The subject technology receives a query, the query including a set of statements, the set of statements including a function call, the function call including a declaration of a vector data type as an argument of the function call. The subject technology processes the query, the processing including invoking the function call. The subject technology provides a set of query results from processing the query, the set of query results including a vector data structure corresponding to the vector data type, the vector data structure including a set of elements, each element comprising a numerical data type.