Data Lake Inventory and Lineage Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Big Data platforms, such as Data Lakes, face challenges in managing vast amounts of data due to the lack of systematic organization, leading to clusters becoming 'data dumps' instead of functional platforms, as users struggle to find and utilize the required data efficiently.
Innovation Solution
Implementing a system that performs automatic inventory and lineage analysis of files, capturing metadata such as format, profile, and provenance, to facilitate systematic organization and access, including methods for lineage discovery, format detection, and parser selection to ensure data usability and compliance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If automatic inventory and lineage analysis are implemented, then data accessibility and organization are improved, but system complexity and computational resources increase
Solution Approach 1:
The system performs preliminary actions by automatically creating inventory records and lineage information for files as they are ingested into the data lake, rather than waiting for users to request information. This proactive approach establishes metadata structures, format profiles, and provenance tracking before users need the data, reducing the complexity burden when users actually access data later.
Solution Approach 2:
The patent introduces an intermediary layer between raw data files and users - the automatic inventory system with metadata repository. This intermediary automatically captures file properties, formats, and lineage information, serving as a mediator that translates complex file structures into user-friendly access points without requiring users to directly navigate the complexity of the underlying data organization.
2Reliability
If comprehensive metadata tracking is implemented, then data integrity is improved, but processing time and storage overhead increase
Solution Approach 1:
The system captures metadata, format information, and lineage data at the time of file ingestion, performing the documentation work preliminarily rather than analyzing it later. This approach ensures data integrity is established from the outset while avoiding repeated processing overhead during user queries, as the information is already captured and stored in the metadata repository.
3Adaptability or versatility
If systematic data organization is implemented, then data usability is improved, but implementation complexity and initial setup time increase
Solution Approach 1:
The system implements self-service by automatically organizing data without requiring manual intervention from users or administrators. The automatic inventory system independently analyzes incoming files, determines their formats, creates metadata records, and establishes lineage relationships, allowing the data lake to self-organize as data is ingested. This eliminates the need for complex manual setup while still achieving systematic organization.
4Manufacturing precision
If lineage discovery and format detection are performed automatically, then compliance and data quality are improved, but computational resources and processing overhead increase
Solution Approach 1:
The system applies partial action by performing format detection and lineage discovery selectively based on file characteristics and user needs. Rather than analyzing every single file in exhaustive detail, the system captures essential metadata and format information at ingestion, and can perform deeper lineage analysis only when necessary for specific queries or compliance requirements, thus balancing data quality with computational resource consumption.
Data Source
AI summary
In system for analyzing large data sets, document/file format can be discovered by attempting to parse the file using several parsers to generate a schema, assigning a score to each parsing, and selecting a parser based on the assigned scores. Schema element attributes, such as statistical parameters, can be derived and used in identifying schema elements associated with other files. Attributes of identified schema elements can be used to substitute missing data values with values based on such attributes. Data values corresponding schema elements can be selected and highlighted, and schema elements and/or attributes thereof can be highlighted based on selected data values. From a cluster of files, a lineage relationship between file pairs, indicating whether one file is derived from another, can be determined for several files. In reducing/compacting data, utilization of all available reducers can be optimized according to current utilization of one or more reducers.