Polymorphic Data Schema Discovery for Relational Transformation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing technologies face challenges in efficiently processing hierarchical, semi-structured data due to its schema-less structure, leading to time-consuming and error-prone data processing, data loss, and inefficiencies in flattening nested structures for reporting and analytics.

Innovation Solution

A system that utilizes a combination of automated processes and human interaction to recursively scan and discover the full schema of hierarchical data, generating metadata for each attribute and data type, allowing users to configure metadata and automatically generate code to transform the data into a fully defined relational format.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If schema-less data formats like JSON are used to store hierarchical data, then flexibility and ease of data storage are improved, but processing time increases and errors become more frequent due to the need for schema discovery

Engineering Contradiction:

Improveflexibility of data formatVSAvoiddata processing efficiency

Core Design Contradiction:

Adaptability or versatilityVSProductivity

Solution Approach 1:

The system performs schema discovery and metadata generation in advance before actual data processing occurs. By pre-defining the schema structure, data types, and relationships through automated scanning and user confirmation, the system eliminates the need for repeated schema discovery during processing, thereby improving processing efficiency while maintaining flexibility

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent introduces metadata as an intermediary layer between the flexible schema-less data format and the processing system. This metadata contains pre-discovered schema information, data types, and relationship definitions that enable efficient processing without requiring the processing system to interpret the flexible format directly, thus resolving the contradiction between format flexibility and processing speed

Inventive Principle:

Principle #24Intermediary (Mediator)

2Measurement precision

If automated schema discovery is performed on hierarchical data, then data type accuracy is improved, but processing time increases due to recursive scanning requirements

Engineering Contradiction:

Improveschema definition accuracyVSAvoidschema discovery time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system performs schema discovery on a sample subset of data first to generate initial metadata, then uses user confirmation to validate and complete the schema definition. This partial action approach achieves sufficient accuracy for processing without requiring exhaustive scanning of entire datasets, thereby reducing time loss while maintaining precision

Inventive Principle:

Principle #16Partial or excessive action

Solution Approach 2:

The system implements a feedback loop where automated schema discovery generates preliminary metadata, which is then presented to users for confirmation and correction. This feedback mechanism ensures high schema accuracy by combining automated detection with human validation, while limiting the time investment to only the automated scanning portion rather than requiring manual review of entire schemas

Inventive Principle:

Principle #23Feedback

3Ease of operation

If nested structures in hierarchical data are flattened for reporting and analytics, then data usability is improved, but data loss may occur due to inadequate flattening methods

Engineering Contradiction:

Improvedata usability for reportingVSAvoiddata quality

Core Design Contradiction:

Ease of operationVSLoss of information

Solution Approach 1:

The system performs preliminary analysis of nested structures during schema discovery to identify relationships, data types, and flattening strategies before actual flattening occurs. By pre-defining how nested elements should be transformed and preserving their relationships in the metadata, the system ensures accurate flattening that maintains data quality while improving usability

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent creates a metadata copy or representation of the nested data structure that preserves the original hierarchy information while enabling flattened views for reporting. This copying approach allows the system to maintain the original data integrity while providing usable flattened representations, thus preventing data loss while improving ease of operation

Inventive Principle:

Principle #26Copying

Data Source

PatentUS12524422B2Systems and methods for processing hierarchical, semi-structured, schema-less, polymorphic data

Publication Date: 2026.01.13 TDAA TECHNOLOGIES CORP

AI summary

Provided are systems and methods to transform a specified source dataset into a fully defined data format that details every attribute value in all levels for each document or record in the source dataset. According to various embodiments, the systems and methods are configured to execute recursive analysis on a specified data source, build out a fully specified data format and associated meta-data, enable user adjustments to a fully schema, and trigger automatic generation of code to transform the source to the new destination format even in the presence of polymorphic data. The process to scan source data involves recursively evaluating every attribute value in all levels for each document or record in the source dataset to account for any polymorphic data. The scan determines each attribute's data type usage, datetime formats and can interrogate string data to identify objects within string data.