Data Lake Hydration With Schema Inference for Retrieval Integrity

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data management systems face challenges in harmonizing diverse data formats, ensuring data quality, and optimizing retrieval and analysis, particularly in large-scale systems, due to the complexities introduced by unstructured, semi-structured, and structured data types, leading to data redundancy, inefficient computational operations, and reduced interoperability.

Innovation Solution

A data lake hydration system with change detection, schema comparison, and in-flight transformation capabilities that normalize and integrate unstructured data, ensuring each update, insertion, or deletion is represented in a unique structured format, using AI algorithms for schema inference and pattern recognition to handle diverse data types in real-time.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If unstructured data is stored directly in the data lake, then data diversity and flexibility are maintained, but data redundancy increases and retrieval efficiency decreases

Engineering Contradiction:

Improvedata format diversityVSAvoidretrieval efficiency

Core Design Contradiction:

Adaptability or versatilityVSProductivity

Solution Approach 1:

The patent segments unstructured data into structured components by extracting entities, attributes, and relationships, organizing them into standardized schemas. This segmentation enables efficient querying and retrieval while preserving the original unstructured data for reference.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces an intermediary layer (structured data representation) between the unstructured data source and the data lake query interface. This intermediary enables efficient retrieval operations without losing the flexibility of unstructured data storage.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If schema validation is performed on all incoming data, then data quality and integrity are improved, but processing time and computational resources increase

Engineering Contradiction:

Improvedata integrityVSAvoidprocessing time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent applies schema validation selectively rather than uniformly to all data. It performs inference-based validation that focuses on critical fields and uses learned patterns from historical data to reduce validation overhead while maintaining data integrity.

Inventive Principle:

Principle #16Partial or excessive action

Solution Approach 2:

The patent performs preliminary schema inference and validation rule generation before actual data validation occurs. By pre-processing and establishing validation criteria in advance, it reduces the computational burden during real-time data processing.

Inventive Principle:

Principle #10Preliminary action

3Stability of the object's composition

If data normalization is performed in real-time, then data consistency across the data lake is maintained, but computational complexity and processing overhead increase

Engineering Contradiction:

Improvedata consistencyVSAvoidcomputational complexity

Core Design Contradiction:

Stability of the object's compositionVSDevice complexity

Solution Approach 1:

The patent implements dynamic normalization that adapts to incoming data characteristics. It uses learned schemas and patterns to perform normalization efficiently in real-time, adjusting the level of normalization based on data type and context rather than applying uniform complex transformations.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The patent changes the parameters of normalization by using inferred schemas and learned data patterns rather than rigid predefined transformations. This allows real-time normalization with reduced computational complexity by adapting to actual data characteristics.

Inventive Principle:

Principle #35Parameter changes

4Productivity

If comprehensive data processing and transformation are performed, then data analytics efficiency is improved, but energy consumption and computational resources increase

Engineering Contradiction:

Improveanalytics efficiencyVSAvoidenergy consumption

Core Design Contradiction:

ProductivityVSUse of energy by moving object

Solution Approach 1:

The patent performs preliminary data processing, transformation, and schema inference before data is loaded into the data lake. By pre-processing data in advance, it reduces the need for complex transformations during analytics operations, thereby improving analytics efficiency while reducing overall energy consumption.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent discards redundant data transformations by establishing schemas and structures upfront, recovering computational resources by avoiding repeated processing of the same data during analytics operations.

Inventive Principle:

Principle #34Discarding and recovering

Data Source

PatentUS20260111416A1Systems and methods for hydrating and maintaining data integrity of a data lake

Publication Date: 2026.04.23 FANNIE MAE

AI summary

Systems, methods, and computer-readable storage media. A system includes one or more processing circuits to identify a modification or update of unstructured data stored in an upstream source, determine a new schema of the unstructured data using a function, the function based on at least one of a pattern, a transformation, an inference, or a correspondence between the unstructured data and a previous schema, determine a difference between the new schema and the previous schema, generate structured data corresponding to the unstructured data, and store the structured data as a representation within the database of the modification or update of the unstructured data.