Metadata-Driven Lakehouse Modeling for Faster Analytical Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional analytical modeling techniques are not ideal for lakehouse data management, as they fail to leverage the advantages of flexible and cost-efficient unstructured/semi-structured data storage, require heavy user involvement, and are not adaptable to evolving business processes or different use cases, leading to slow modeling and scalability issues.
Innovation Solution
Implement metadata-driven database denormalization and modeling techniques, including schema analysis, query statistics, and transformation jobs to optimize data structure for lakehouses, using methods like path, edge, tree, log, and adaptive denormalization, and state machine modeling to transform data from operational databases to lakehouses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional analytical modeling techniques are used for lakehouse data management, then data can be stored in structured format, but query performance is slow and data access efficiency is poor
Solution Approach 1:
The patent implements adaptive denormalization that dynamically adjusts data structure based on query patterns and workload characteristics. The system automatically transforms normalized data into denormalized formats when analytical queries are detected, and re-normalizes when transactional workloads dominate, optimizing query performance without manual intervention.
Solution Approach 2:
The system changes the structural parameters of data storage by implementing multiple denormalization strategies (path, edge, tree, log, and adaptive denormalization) that transform data from highly normalized relational structures into denormalized formats suitable for lakehouse analytics, improving query performance by reducing join operations.
2Adaptability or versatility
If manual analytical modeling is performed, then data models can be customized for specific use cases, but the process is slow and not adaptable to evolving business processes
Solution Approach 1:
The system implements self-service automated analytical modeling that automatically generates and transforms data models based on detected query patterns and business requirements. The automated metadata-driven transformation engine continuously adapts data structures without requiring manual analyst intervention, enabling rapid response to evolving business processes.
Solution Approach 2:
The system incorporates feedback loops that monitor query patterns, workload characteristics, and data access patterns to automatically adjust and refine data models. This continuous feedback mechanism enables the system to adapt to changing business requirements and optimize data structures over time without manual reconfiguration.
3Productivity
If normalized data structure is maintained in lakehouse, then data storage is efficient, but query performance deteriorates due to multiple joins required
Solution Approach 1:
The patent implements dynamic data structure transformation that automatically denormalizes data from normalized relational formats into lakehouse-optimized denormalized structures when analytical queries are detected. This dynamic transformation maintains storage efficiency while eliminating the need for multiple join operations during query execution, thereby improving query performance.
4Manufacturing precision
If heavy user involvement is required for data modeling, then complex analytical requirements can be addressed, but the process becomes cumbersome and difficult to scale
Solution Approach 1:
The system implements self-service automated analytical modeling that generates precise data models automatically based on detected business requirements and query patterns. The metadata-driven transformation engine handles complex modeling tasks without manual intervention, maintaining high analytical precision while eliminating the complexity and labor intensity of manual modeling processes.
Data Source
AI summary
Systems and methods are provided for transforming data stored in an operational database to a format/structure optimized for use in a data lakehouse. The data transformation is metadata-driven, where the metadata characterizing the transformation of data may be automatically generated via the performance of various denormalization/modeling techniques, including: path/edge/tree/log denormalization, and state machine/aggregate/adaptive modeling.