Lakehouse Dataset View Generation by Query-Table Similarity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In a data lakehouse, users face difficulty in accessing the right dataset for their intended purpose due to the diversity of datasets, making it challenging to form a dataset that meets their requirements.
Innovation Solution
A computer-implemented method that computes similarity between user queries and table descriptions in a data lakehouse, selects relevant tables based on threshold values, joins them to create a new view, and generates a description of the dataset using semantic annotations and embeddings, ultimately displaying it to the user if the similarity exceeds predefined thresholds.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If users manually search through diverse datasets in a data lakehouse, then they can access any data, but the complexity and time required to find the right dataset increases significantly
Solution Approach 1:
The patent introduces an intermediary system consisting of a query processor and similarity computation module that mediates between user queries and datasets. This intermediary automatically computes similarity scores between user queries and dataset descriptions, filtering and ranking datasets based on relevance without requiring users to manually search through the entire data lakehouse.
Solution Approach 2:
The patent replaces the mechanical manual search process with an automated computational system. Instead of users manually browsing through diverse datasets, the system automatically computes similarity metrics between queries and datasets using computational algorithms, substituting human manual operations with automated mechanical processes.
2Adaptability or versatility
If the data lakehouse stores diverse datasets for flexibility, then it can support various data analytics and AI workloads, but it becomes difficult for users to access the right dataset
Solution Approach 1:
The patent implements a universal query processing mechanism that handles diverse datasets through a single similarity computation framework. The system can process different types of queries (text, structured, unstructured) and match them against various dataset types using the same automated similarity assessment mechanism, making the system universally applicable across different data analytics and AI workloads.
Solution Approach 2:
The query processor acts as an intermediary layer that sits between users and the diverse datasets. It translates user queries into computable similarity assessments and automatically filters datasets based on relevance, maintaining the versatility of the data lakehouse while simplifying user access to the right data.
3Adaptability or versatility
If multiple tables are joined to form a comprehensive view, then the data becomes more useful, but the complexity of data navigation and selection increases
Solution Approach 1:
The system performs preliminary action by pre-computing and storing descriptions of datasets and their relationships before user queries arrive. When a query comes in, the system can immediately compute similarity scores against the pre-existing dataset descriptions and metadata, avoiding the need for users to manually navigate through complex table relationships and joins.
Solution Approach 2:
The patent introduces an intermediary description generation mechanism that creates simplified representations of complex table joins and relationships. This intermediary layer translates complex multi-table joins into manageable dataset descriptions that can be easily compared against user queries, reducing the perceived complexity while maintaining data completeness.
Data Source
AI summary
Described are techniques for generating a query based view of appropriate datasets in a data lakehouse. A query for data contained in the data lakehouse is received. A similarity between the query and a description of each table in the lakehouse is computed. Any description of a table with a similarity to the query that is greater than a threshold value results in the table potentially being included in a view. As a result, such a table is selected. A subset of the selected tables is then identified and joined to create a new view of a dataset. After generating a description of the new view based on headers and values of the dataset, a similarity between the query and the description of the new view is computed. If the similarity measure exceeds a threshold value, then the newly constructed view is displayed to the user.