Data Lake Hydration with Schema Divergence Normalization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data management systems struggle with integrating diverse data formats in data lakes, leading to data redundancy, inefficiencies, and computational challenges due to unstructured and semi-structured data, especially in large-scale environments.
Innovation Solution
A data lake hydration system with change detection, schema comparison, and in-flight transformation capabilities normalizes unstructured data in real-time, ensuring unique structured representations and reducing duplicates, while adapting to diverse data types using AI and ML algorithms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If unstructured data is stored in a data lake without normalization, then data storage flexibility is improved, but data redundancy increases and computational efficiency deteriorates
Solution Approach 1:
The patent segments the data normalization process into multiple stages: schema inference, schema comparison, data transformation, and validation. Each stage handles a specific aspect of normalization, allowing the system to process diverse unstructured data into standardized formats while minimizing redundancy through systematic decomposition of the complex normalization task.
Solution Approach 2:
The system performs preliminary schema inference and schema comparison before actual data transformation. By determining the schema of incoming unstructured data and comparing it against existing schemas in advance, the system prepares transformation rules ahead of time, enabling efficient normalization without redundant processing and reducing computational overhead.
2Reliability
If real-time data normalization is performed, then data integrity is improved, but computational resources and processing time increase
Solution Approach 1:
The system applies partial normalization by selectively transforming only the portions of unstructured data that require schema mapping, rather than normalizing entire datasets. The schema comparison mechanism identifies only the divergent elements between new and existing schemas, applying transformation only where necessary, thus reducing computational resource consumption while maintaining data integrity.
Solution Approach 2:
The system changes parameters dynamically by inferring schemas from incoming data and adjusting transformation rules based on schema comparisons. This adaptive approach allows the normalization process to optimize its computational parameters based on the actual data characteristics, reducing unnecessary processing while ensuring complete normalization for data integrity.
3Adaptability or versatility
If schema comparison and transformation functions are added, then data interoperability is improved, but system complexity increases
Solution Approach 1:
The patent implements a universal schema inference and comparison framework that handles multiple data types and formats through a single integrated system. The same infrastructure processes diverse schemas by inferring their structures, comparing them against existing schemas, and applying appropriate transformations, thereby achieving broad data interoperability without requiring separate specialized systems for each data type.
Solution Approach 2:
The system introduces schema inference and comparison as intermediary layers between data sources and the data lake. These intermediary functions act as mediators that translate diverse unstructured data into standardized formats, absorbing the complexity of schema variations and presenting a unified interface to downstream systems, thus improving interoperability while isolating complexity in dedicated transformation layers.
4Measurement precision
If comprehensive schema inference and comparison processes are implemented, then data accuracy is improved, but processing speed decreases
Solution Approach 1:
The system performs schema inference and schema comparison as preliminary actions before data transformation. By determining the schema and identifying divergences in advance, the system prepares precise transformation rules that ensure accurate data normalization. This preliminary analysis enables faster subsequent transformation operations while maintaining high data accuracy through comprehensive schema understanding.
Solution Approach 2:
The schema comparison process provides feedback about schema divergences and transformation requirements back to the data transformation stage. This feedback mechanism ensures that transformation functions are precisely tailored to the actual schema differences detected, improving data accuracy. The feedback loop also enables optimization of processing speed by adjusting transformation strategies based on the complexity and characteristics of detected schema variations.
Data Source
AI summary
Disclosed are systems and methods for maintaining data integrity (e.g., of a database, a data lake, etc.). In some embodiments, one or more processing circuits identify an update, insertion, or deletion of unstructured data stored in an upstream source. The processing circuits determine a new schema of the unstructured data using an inferring function and further determine a divergence between the new schema and a previous schema of the unstructured data based on comparing the new schema to the previous schema of the unstructured data. In an example, the processing circuits store the structured data in a database. The database includes an established data channel between the database and a downstream source, and the structured data is a unique representation within the database of the update, insertion, or deletion of the unstructured data.