Parallelized Database Indexing via Data Lakehouse Object Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face limitations in processing speed due to hardware constraints, data storage methods, and restricted co-processing options, which hinder efficient query execution and data retrieval.
Innovation Solution
The implementation of a large-scale data processing network with a parallelized database system that utilizes a query execution plan across multiple nodes, incorporating index structures and object storage systems to optimize data access and processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in traditional database systems with centralized processing, then data consistency and reliability are maintained, but processing speed and query execution efficiency deteriorate due to hardware constraints and single-point bottlenecks
Solution Approach 1:
The patent divides the database system into multiple distributed nodes, each capable of independent query processing. Data is partitioned across these nodes, allowing parallel execution of query operations. This segmentation eliminates centralized processing bottlenecks and improves overall processing speed while maintaining data consistency through distributed coordination protocols.
Solution Approach 2:
The patent transitions from traditional single-dimension storage to multi-dimensional data organization by implementing columnar storage formats and creating data lakes that allow queries to access data from multiple perspectives. This dimensional transformation enables more efficient query execution by organizing data according to access patterns rather than transactional sequences.
2Productivity
If traditional storage methods are used, then implementation simplicity is maintained, but query execution efficiency and data retrieval speed deteriorate
Solution Approach 1:
The patent implements pre-computed indexes and materialized views that are created in advance to accelerate query execution. By performing data preparation and indexing operations beforehand, the system reduces the computational burden during actual query execution, significantly improving productivity without adding complex runtime processing logic.
Solution Approach 2:
The patent introduces a query optimization layer that acts as an intermediary between the user and the distributed storage system. This layer handles query parsing, optimization, and execution plan generation, shielding users from the complexity of distributed data access while maximizing query execution efficiency through intelligent query rewriting and resource allocation.
3Adaptability or versatility
If co-processing options are restricted in traditional systems, then system simplicity is maintained, but processing capability and scalability deteriorate
Solution Approach 1:
The patent creates a universal data processing platform that can handle multiple types of workloads including analytical queries, transactional operations, and machine learning inference within the same distributed architecture. This multi-functional system allows diverse co-processing capabilities while maintaining a unified management interface that reduces configuration complexity.
Solution Approach 2:
The patent implements self-service capabilities where the distributed database system automatically performs query optimization, resource allocation, and load balancing without requiring manual configuration. The system adapts to varying workloads dynamically, providing enhanced co-processing capability while keeping the user experience simple through automated resource management.
Data Source
AI summary
A data storage system is operable to storing a first plurality of files and a second plurality of files in memory resources of an object storage system of a data storage system. The first plurality of files store a plurality of records of at least one table, and the second plurality of files store a set of index structures indexing the plurality of records. Table metadata is generated for storage, mapping the first plurality of files and the second plurality of files to the at least one table, via a metadata processing system of the data storage system. A filtered row set identifying a proper subset of the plurality of records meeting filtering parameter data is generated based on accessing the table metadata, and based on further accessing at least one file of the second plurality of files in the object storage system storing at least one index structure.