Data Lake Domain Indexing via Locality-Sensitive Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data management tools fail to provide a unified view of data in data lakes, leading to complex tasks in data discovery and integration due to the difficulty in understanding relationships between files and identifying relevant data sources.
Innovation Solution
A method that involves obtaining data files from a data lake, selecting candidate fields based on record types, determining relevance scores using natural language processing and ontology resources, and clustering similar domains using the Locality-Sensitive-Hashing algorithm to integrate data files based on corresponding domains, facilitating data integration and exploratory data analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data files are stored in original form in a data lake, then data volume and storage capacity are improved, but data discovery and integration become complex and difficult
Solution Approach 1:
The system performs preliminary actions by automatically generating metadata, extracting schemas, and creating data catalogs before users need to discover or integrate data. This advance preparation reduces the complexity of data integration when users later access the data lake, as the heavy lifting of understanding data structures has already been done.
Solution Approach 2:
The patent introduces an intermediary layer between the raw data files and users - specifically a data catalog and metadata system that mediates between the stored data and discovery needs. This intermediary provides unified views, schema information, and relationship mappings without requiring users to directly navigate the complex raw data structures.
2Measurement precision
If manual data integration methods are used, then data accuracy and understanding are improved, but time consumption and productivity are reduced
Solution Approach 1:
The system enables self-service by automatically performing data discovery, schema extraction, metadata generation, and integration task creation without requiring manual user intervention for each step. The automated agents and systems work independently to understand and integrate data, dramatically improving productivity while maintaining accuracy through systematic approaches.
Solution Approach 2:
The patent changes parameters by transforming unstructured or semi-structured data into structured formats with defined schemas, metadata, and standardized representations. This parameter transformation maintains data understanding accuracy by preserving semantic meaning while enabling automated processing and faster integration through standardized data formats.
3Loss of information
If comprehensive data analysis is performed on all files, then data discovery completeness is improved, but processing time and computational resources are reduced
Solution Approach 1:
The system applies partial action by focusing computational resources on extracting only the most relevant metadata, schemas, and relationships needed for data discovery, rather than performing exhaustive analysis of every data file in its entirety. This selective approach maintains sufficient discovery completeness while dramatically reducing processing time and resource consumption.
Solution Approach 2:
The patent segments the data analysis process into distinct components: metadata extraction, schema generation, relationship identification, and catalog creation. This segmentation allows the system to process different aspects of data files independently and in parallel, improving efficiency while maintaining comprehensive data discovery through systematic coverage of all necessary elements.
Data Source
AI summary
Techniques are provided for data discovery and data integration in a data lake. One method comprises obtaining data files from a data lake, wherein each data file comprises multiple records having multiple fields; selecting multiple candidate fields from a data file based on a record type; determining a relevance score for each candidate field from the data file based on multiple features extracted from the data file; and clustering the scored candidate fields into clusters of similar domains using a hashing algorithm, wherein a given cluster comprises candidate fields, wherein multiple data files can be integrated based on a domain of the candidate fields in the given cluster. The relevance score for each candidate field is based on multiple features comprising, for example, features that take into account a morphological or semantic similarity between file name, file metadata and/or file records and features that consider statistics of candidate fields in a data file.