Data Lake File Lineage and Format Discovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Big Data platforms like Data Lakes face challenges in managing vast amounts of data due to the lack of systematic organization, leading to clusters becoming 'data dumps' instead of functional platforms, as users struggle to find and utilize the required data effectively.
Innovation Solution
Implementing a system that performs automatic inventory and lineage analysis of files, capturing metadata such as format, profile, and provenance, to facilitate systematic organization and access, including methods for format discovery, parser selection, and lineage tracing to determine file purpose and origin.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If automatic inventory and lineage analysis systems are implemented, then data accessibility and organization improve, but system complexity and implementation cost increase
Solution Approach 1:
The system performs automatic self-inventory and self-organization of data files through automated lineage analysis. The metadata repository automatically captures file relationships, formats, and provenance information without requiring manual intervention, enabling the system to serve itself in maintaining organizational structure while improving data accessibility to users.
Solution Approach 2:
A metadata repository acts as an intermediary layer between the raw data files and users. This intermediary automatically captures and stores lineage information, file formats, and organizational relationships, allowing users to access data through simplified interfaces without directly navigating the complex underlying file structure.
2Reliability
If comprehensive metadata capture and lineage tracking are implemented, then data integrity and compliance improve, but processing time and computational resources increase
Solution Approach 1:
The system performs preliminary metadata capture and lineage tracking at the time of file ingestion and creation events. By capturing organizational structure, file relationships, and format information proactively when files are first introduced to the data lake, the system establishes integrity records in advance rather than requiring retrospective analysis, thereby reducing subsequent processing time while maintaining comprehensive data integrity.
3Productivity
If systematic data organization and lineage analysis are implemented, then data retrieval efficiency improves, but storage overhead and infrastructure cost increase
Solution Approach 1:
The system creates lightweight metadata copies and lineage representations of data files rather than duplicating the actual data content. The metadata repository stores organizational structure, file relationships, and descriptive information as compact data structures that reference original files, enabling efficient data retrieval through metadata queries without requiring substantial additional storage capacity.
Data Source
AI summary
In system for analyzing large data sets, document/file format can be discovered by attempting to parse the file using several parsers to generate a schema, assigning a score to each parsing, and selecting a parser based on the assigned scores. Schema element attributes, such as statistical parameters, can be derived and used in identifying schema elements associated with other files. Attributes of identified schema elements can be used to substitute missing data values with values based on such attributes. Data values corresponding schema elements can be selected and highlighted, and schema elements and/or attributes thereof can be highlighted based on selected data values. From a cluster of files, a lineage relationship between file pairs, indicating whether one file is derived from another, can be determined for several files. In reducing/compacting data, utilization of all available reducers can be optimized according to current utilization of one or more reducers.