Columnar Metadata Management for Big Data Query Performance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
As data volumes grow, traditional big data systems face performance degradation due to increasing metadata sizes, requiring a method to manage large quantities of metadata without reducing its amount, which is essential for scalability and performance.
Innovation Solution
The system employs columnar techniques to manage big metadata by generating a system query over associated system tables, filtering data blocks using metadata, and generating a final query that scans only the necessary data blocks, thereby reducing processing time and latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional big data systems are used to manage large datasets, then data storage capacity is maintained, but query performance degrades due to increasing metadata sizes
Solution Approach 1:
The patent segments the metadata management by introducing system tables that store metadata in a structured, columnar format. This segmentation allows the system to efficiently query and filter metadata without scanning the entire metadata set, thereby maintaining query performance even as data storage capacity increases.
Solution Approach 2:
The patent introduces a new dimension for metadata organization by using system tables with columnar storage. This dimensional change from traditional flat metadata storage to structured system tables enables efficient filtering and querying, resolving the performance degradation issue while maintaining large storage capacity.
2Loss of information
If metadata amount increases to support larger data tables, then data locability is improved, but system performance degrades
Solution Approach 1:
The patent applies preliminary action by pre-organizing metadata into system tables with columnar storage before queries are executed. This preliminary structuring enables efficient filtering and retrieval operations, allowing the system to maintain high performance even with increased metadata amounts required for better data locability.
3Measurement precision
If all data blocks are scanned to ensure query accuracy, then measurement precision is maintained, but processing time increases significantly
Solution Approach 1:
The patent extracts only the necessary data blocks for querying by using system tables to pre-filter and identify relevant blocks. This extraction approach maintains query accuracy by ensuring all necessary blocks are retrieved while eliminating unnecessary scanning, thereby significantly reducing processing time.
Solution Approach 2:
The patent applies partial action by scanning only the subset of data blocks identified as potentially relevant through system table queries, rather than scanning all data blocks. This partial scanning approach maintains sufficient query accuracy while dramatically reducing processing time.
Data Source
AI summary
A method for managing big metadata using columnar techniques includes receiving a query request requesting data blocks from a data table that match query parameters. The data table is associated with system tables that each includes metadata for a corresponding data block of the data table. The method includes generating, based on the query request, a system query to return a subset of rows that correspond to the data blocks that match the query parameters. The method further includes generating, based on the query request and the system query, a final query to return a subset of data blocks from the data table corresponding to the subset of rows. The method also includes determining whether any of the data blocks in the subset of data blocks match the query parameters, and returning the matching data blocks when one or more data blocks match the query parameters.