Business Data Lake Search Engine Using Graph-Based Relevance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current Business Data Lake search tools fail to effectively retrieve relevant data assets due to limitations in handling semantic similarities, inter-relations among data assets, query history, user profiles, and morphological variations, leading to inefficient search results for users.
Innovation Solution
A method involving a directed graphical structure representing tables in the Business Data Lake, using node rank algorithms, NLP-based models, and collaborative filtering to rank and enrich search results based on relevancy scores, user interactions, and semantic similarities, thereby improving search accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If primitive textual search tools are used, then search implementation is simple, but search relevance is poor
Solution Approach 1:
The patent combines multiple search methodologies including term frequency-based scoring, node rank algorithms for graph-based relevance, topic models for semantic understanding, and collaborative filtering for user behavior analysis. These diverse approaches are merged into a unified search system that processes queries through multiple stages, integrating results from different algorithms to produce comprehensive and accurate search results that address the limitation of simple textual search tools.
2Speed
If only keyword matching is used, then search processing is fast, but semantic similarity is not captured
Solution Approach 1:
The patent performs preliminary processing by pre-computing topic models for data assets and pre-building graph representations of relationships between assets. These pre-computed structures enable rapid query processing while capturing semantic similarities through established models rather than computing relationships in real-time, thus maintaining speed while improving semantic understanding.
Solution Approach 2:
The patent introduces topic models and graph representations as intermediary structures between the query and the actual data assets. These intermediaries translate user queries into semantic representations and facilitate efficient matching with relevant assets, enabling both fast processing and accurate semantic similarity capture without direct complex computation during query time.
3Device complexity
If data asset relationships are ignored, then search structure is simple, but inter-relations among data assets are not considered
Solution Approach 1:
The patent segments the search process into distinct phases: initial keyword matching, graph-based relationship analysis, topic model semantic matching, and collaborative filtering refinement. Each phase handles specific aspects of relationship analysis separately, allowing the system to consider inter-relations among data assets through modular processing stages rather than a monolithic complex structure.
4Device complexity
If user profile and query history are not used, then search system is simple, but personalized results are not provided
Solution Approach 1:
The patent implements feedback mechanisms through collaborative filtering that analyzes user interactions with search results and adjusts future search recommendations accordingly. User behavior patterns are continuously monitored and fed back into the search system to refine personalization, enabling adaptability and versatility in providing tailored results while managing system complexity through iterative improvement rather than overly complex pre-configured personalization structures.
Data Source
AI summary
Business Data Lake searching techniques are provided. A method comprises obtaining a graph representing tables of the Business Data Lake, where each node represents one table and edges between nodes represent foreign key connections; applying a node rank algorithm to determine a relevancy score of the tables based on a number of links to/from other tables; and, in response to a query: ranking a relevancy of query items based on a term frequency-based score to generate candidate results; extracting a candidate sub-graph based on the following: a top-L tables based on the term frequency-based score, and/or a top-M tables based on a topic model distance score for the given query and candidate items; enriching the extracted candidate sub-graph by adding new tables using an item-to-item collaborative filter where a similarity between two tables is measured based on a number of interactions; and ordering the tables in the enriched sub-graph based on the relevancy score and/or a user-to-item collaborative filter that evaluates past user interactions with prior results.