Universal Data Lake Indexing for Multi-Engine Query Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge of efficiently indexing and querying large volumes of unstructured data in data lakes, particularly in distributed and cloud-based systems, is exacerbated by the lack of common indexes across disparate query engines, hindering performance optimization.
Innovation Solution
A system and method for creating, discovering, and using indexes on a data lake that conform to an index specification, enabling multi-engine data workload optimization through index metadata management and serverless index operations, allowing integration with various query engines.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If indexing solutions are provided in distributed database systems and cloud-based architectures, then query performance is improved, but compatibility across disparate query engines deteriorates
Solution Approach 1:
The patent creates a universal index specification that enables indexes to be used across multiple disparate query engines (Spark, Presto, Trino, Flink, etc.). The index specification serves as a common interface that different query engines can understand and utilize, allowing a single index to provide performance benefits across multiple engines rather than requiring separate indexes for each engine.
Solution Approach 2:
The patent introduces an intermediary layer (the standardized index specification and catalog) that mediates between the index implementation and various query engines. This intermediary translates and adapts index information into formats that different query engines can understand, enabling compatibility without requiring each engine to implement its own proprietary indexing system.
2Adaptability or versatility
If common indexes are made accessible to multiple query engines, then multi-engine interoperability is improved, but index management complexity increases
Solution Approach 1:
The patent implements self-service mechanisms where the system automatically discovers, validates, and manages indexes through standardized APIs. The index catalog automatically tracks which indexes are available to which query engines, and the system handles index creation, validation, and lifecycle management without requiring manual intervention for each query engine integration.
Solution Approach 2:
The patent uses parameter-based index specifications that can be configured and adapted through standardized parameters rather than complex structural changes. By managing indexes through configurable parameters and metadata in a catalog, the system simplifies management while maintaining multi-engine compatibility.
3Productivity
If indexes are created to optimize query performance, then data retrieval efficiency is improved, but storage requirements increase
Solution Approach 1:
The patent merges index data with existing data lake storage infrastructure rather than maintaining separate dedicated index storage systems. By storing indexes in the data lake alongside the source data and utilizing existing storage capacity, the system minimizes additional storage requirements while still providing the performance benefits of indexing.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods, systems and computer program products are described herein that provide a serverless, multi-engine, multi-user data lake indexing subsystem and application programming interface. Indexes are defined as derived datasets and stored on the data lake in a universal format that enables disparate engines to create and/or discover indexes for workload optimization. Embodiment of indexes enable stateful control and management of an index via metadata included in the index and stored on the data lake.