Query Predicate Derivation Using Metadata for Faster Data Pruning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional database management systems require significant computing and storage resources, have limited scalability, and are costly, with cloud-based solutions offering limited metadata handling capabilities, leading to inefficiencies in data access and management.
Innovation Solution
Implementing a system with micro-partitions for database data and metadata storage in immutable cloud storage, utilizing a resource manager to organize and manage metadata across multiple levels, enabling efficient pruning and query response through granular selection of relevant data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional database management systems are used, then data can be stored and managed, but significant computing and storage resources are required and scalability is limited
Solution Approach 1:
The patent segments metadata into multiple hierarchical levels (table-level, column-level, row-level) and stores them in immutable cloud storage. This segmentation allows the system to manage large volumes of metadata efficiently without requiring proportional computing resources, as each level can be independently pruned and accessed based on query needs.
Solution Approach 2:
The patent extracts metadata handling from the traditional database management system and stores it separately in immutable cloud storage. This extraction eliminates the need for the database system to maintain metadata in volatile memory, significantly reducing computing and storage resource requirements while improving scalability.
2Adaptability or versatility
If cloud-based database systems are used, then data accessibility and scalability improve, but metadata handling capabilities remain limited
Solution Approach 1:
The patent adds a hierarchical dimension to metadata organization by creating multiple levels of metadata (table-level, column-level, row-level) that can be selectively accessed. This dimensional approach to metadata management enhances cloud-based systems' ability to handle complex queries while maintaining data accessibility across distributed environments.
3Speed
If extensive metadata is stored to improve query optimization, then query performance improves, but storage resources and retrieval time increase
Solution Approach 1:
The patent applies local quality by storing different levels of metadata detail in different locations within the immutable storage system. Frequently accessed metadata is stored at higher hierarchical levels for quick retrieval, while detailed metadata is stored at lower levels and only accessed when needed for query optimization, reducing overall storage requirements while maintaining query performance.
4Reliability
If traditional database architecture is used, then data can be managed centrally, but significant physical space and capital investment are required
Solution Approach 1:
The patent introduces immutable cloud storage as an intermediary between the database system and physical storage infrastructure. This intermediary layer allows the database to manage data centrally with full reliability while offloading the physical storage burden to the cloud provider, eliminating the need for significant on-premises physical infrastructure.
Data Source
AI summary
The subject technology receives a first query plan, the first query plan including a set of statements, the set of statements including at least one statement with at least a reference to a particular column in a first table. The subject technology identifies a first predicate from the at least one statement, the first predicate corresponding to a join operation. The subject technology determines, for the particular column, a set of expression properties. The subject technology, based on the set of expression properties, determines a first derived predicate, the first derived predicate corresponding to a first filter operation. The subject technology generates a new operation based on the first derived predicate. The subject technology generates a new query plan based at least in part on the join operation of first predicate, the first derived predicate, and the new operation of the first derived predicate.