Recursive Schema Discovery for Polymorphic Hierarchical Data

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing technologies face challenges in efficiently processing hierarchical, semi-structured data due to its schema-less structure, leading to time-consuming and error-prone data processing, data loss, and difficulties in flattening nested structures for reporting and analytics.

Innovation Solution

A system and method that employs a combination of automated processes and human interaction to recursively scan and discover the full schema of hierarchical data, generating metadata for each attribute, allowing users to configure and modify metadata, and automatically generating code to transform the data into a fully defined relational format.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If hierarchical, semi-structured data is processed using conventional methods, then the data can be stored and transferred flexibly, but the processing becomes time-consuming and error-prone due to schema-less structure

Engineering Contradiction:

Improvedata format flexibilityVSAvoiddata processing efficiency

Core Design Contradiction:

Adaptability or versatilityVSProductivity

Solution Approach 1:

The system performs preliminary schema discovery and metadata generation before actual data processing. By scanning the hierarchical data source and generating a complete schema definition upfront, the system prepares all necessary processing rules in advance, eliminating the need for repeated schema analysis during data transformation operations.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system introduces metadata as an intermediary layer between the hierarchical data source and the processing logic. This metadata contains the discovered schema information and serves as a bridge that enables efficient processing without requiring the processing engine to interpret the complex hierarchical structure directly.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If the schema is discovered and processed manually, then data accuracy can be maintained, but additional engineering work is required each time the schema evolves

Engineering Contradiction:

Improvedata accuracyVSAvoidengineering work requirements

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The system performs self-service schema discovery by automatically scanning the hierarchical data source and generating metadata without human intervention. The schema evolution is handled automatically through recurring scans that detect changes and update the metadata, eliminating the need for manual schema management whenever the data structure changes.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system implements feedback mechanisms through recurring scans that monitor the hierarchical data source for schema changes. When changes are detected, the system automatically updates the metadata and notifies relevant components, ensuring data accuracy is maintained through continuous validation against the updated schema.

Inventive Principle:

Principle #23Feedback

3Ease of operation

If nested structures are flattened using conventional methods, then data can be used in reporting and analytics, but data loss can occur leading to degraded data quality

Engineering Contradiction:

Improvedata usability in reportingVSAvoiddata quality

Core Design Contradiction:

Ease of operationVSReliability

Solution Approach 1:

The system performs preliminary flattening operations during the metadata generation phase, creating a complete flattened representation of the hierarchical data before actual processing. This ensures that all nested structures are properly resolved and data is ready for reporting and analytics without risking data loss during subsequent operations.

Inventive Principle:

Principle #10Preliminary action

4Productivity

If a single pass extraction and flattening is performed, then processing speed can be improved, but the polymorphic nature of the data is not adequately handled

Engineering Contradiction:

Improveprocessing speedVSAvoidschema discovery accuracy

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The system performs preliminary schema discovery through recurring scans before the actual extraction and flattening operations. By completing the schema analysis upfront and generating comprehensive metadata, the system enables a single-pass data extraction process that maintains high speed while ensuring complete and accurate schema handling through the pre-prepared metadata.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS20260099504A1Systems and methods for processing hierarchical, semi-structured, schema-less, polymorphic data

Publication Date: 2026.04.09 TDAA TECHNOLOGIES CORP

AI summary

Provided are systems and methods to transform a specified source dataset into a fully defined data format that details every attribute value in all levels for each document or record in the source dataset. According to various embodiments, the systems and methods are configured to execute recursive analysis on a specified data source, build out a fully specified data format and associated meta-data, enable user adjustments to a fully schema, and trigger automatic generation of code to transform the source to the new destination format even in the presence of polymorphic data. The process to scan source data involves recursively evaluating every attribute value in all levels for each document or record in the source dataset to account for any polymorphic data. The scan determines each attribute's data type usage, datetime formats and can interrogate string data to identify objects within string data.