Dynamic Hierarchical Data Flow Mapping for Structural Variability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data flow mapping systems require redundant schema definitions for hierarchical data, leading to operational overhead and inefficient use of resources when processing data from multiple sources with structural differences, necessitating separate mappings for minor variations.
Innovation Solution
A dynamic hierarchical data flow mapping system that allows for schema-less descriptions of hierarchical data, enabling a single reusable data flow mapping across sources with structural differences, using dynamic hierarchical fields that adapt during compilation and execution, and leveraging rule-based transformations to handle varying column schemas.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If separate data flow mappings are defined for each data source with structural differences, then data processing accuracy is improved, but operational overhead and device complexity increase significantly
Solution Approach 1:
The patent implements a universal data flow mapping system that can process hierarchical data from multiple data sources with different structures using a single mapping definition. The system achieves this by dynamically generating schema definitions at runtime based on the actual data structure, allowing one mapping to serve multiple sources without requiring separate mappings for each source variant.
Solution Approach 2:
The system employs dynamic schema generation where the data flow mapping adapts to different data source structures at runtime. Instead of static schema definitions, the system dynamically determines the hierarchical structure of incoming data and generates appropriate schema definitions on-the-fly, enabling a single mapping to handle multiple structural variations.
2Manufacturing precision
If separate data flow mappings are defined for each data source with structural differences, then data processing accuracy is improved, but computing resource usage increases inefficiently
Solution Approach 1:
The patent implements a universal data flow mapping system that can process hierarchical data from multiple data sources with different structures using a single mapping definition. The system achieves this by dynamically generating schema definitions at runtime based on the actual data structure, allowing one mapping to serve multiple sources without requiring separate mappings for each source variant.
Solution Approach 2:
The system changes the parameter of schema definition from static to dynamic. By generating schema definitions dynamically based on the actual data structure at runtime, the system avoids the computational overhead of maintaining and processing multiple static schema definitions, thereby reducing computing resource usage while maintaining accurate data processing.
3Reliability
If redundant schema definitions are created for hierarchical columns in each data flow mapping, then data structure compatibility is improved, but loss of time in maintenance and operational overhead increase
Solution Approach 1:
The system employs dynamic schema generation where the data flow mapping adapts to different data source structures at runtime. Instead of static schema definitions, the system dynamically determines the hierarchical structure of incoming data and generates appropriate schema definitions on-the-fly, enabling a single mapping to handle multiple structural variations.
Solution Approach 2:
The system implements self-service schema generation by automatically determining the hierarchical structure of incoming data and generating appropriate schema definitions without requiring manual intervention. This automated approach eliminates the time-consuming maintenance of redundant schema definitions while ensuring data structure compatibility.
4Device complexity
If a single reusable data flow mapping is used across multiple sources, then operational overhead is reduced, but difficulty in handling structural differences increases
Solution Approach 1:
The system employs dynamic schema generation where the data flow mapping adapts to different data source structures at runtime. Instead of static schema definitions, the system dynamically determines the hierarchical structure of incoming data and generates appropriate schema definitions on-the-fly, enabling a single mapping to handle multiple structural variations.
Solution Approach 2:
The system introduces an intermediary dynamic schema generation layer between the single reusable data flow mapping and the diverse data sources. This intermediary automatically adapts the mapping to handle structural differences by generating appropriate schema definitions, thereby simplifying operational overhead while managing structural variability.
Data Source
AI summary
A method, apparatus, and computer-readable medium for dynamic hierarchical data flow mapping, including storing dynamic data flow mappings, each dynamic data flow mapping including a mapping of data from an input port corresponding to a source data container to an output port corresponding to a destination data container and a dynamic hierarchical field having a dynamic hierarchical data type, determining a rule language corresponding to each dynamic hierarchical field in the dynamic data flow mappings, the rule language defining acceptable parameters for sub-fields of that dynamic hierarchical data type, and generating static data flow mappings by resolving all dynamic hierarchical fields into static hierarchical fields during compilation of the dynamic data flow mappings with the source data container and the destination data container, the static data flow mappings being generated based at least in part on the rule language corresponding to each dynamic hierarchical field and underlying data in one or more of the source data container or the destination data container.