Data Model Entity Mapping via Automated Dataflow Graph Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems for performing complex computations on data models, such as those involving data flow through directed graphs, require manual construction of dataflow graphs and lack efficient methods for mapping between source and destination schemas, especially when auxiliary information from external sources is involved.
Innovation Solution
The system generates a procedural specification, in the form of an executable dataflow graph, by enabling users to specify mappings between source and destination schemas at a high level of abstraction, automatically breaking down problem statements into joins, rollups, and transformations, and referencing auxiliary information from external databases or files.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If manual construction of dataflow graphs is performed, then precision in data model mapping is improved, but productivity deteriorates due to increased manual effort and time consumption
Solution Approach 1:
The system performs preliminary analysis of source and destination schemas to automatically generate dataflow graphs before manual intervention is needed. By pre-computing the mapping structure and transformation logic, the system reduces both manual effort and maintains accuracy, resolving the contradiction between precision and productivity
Solution Approach 2:
The system enables self-service automated generation of dataflow graphs by analyzing schema definitions and generating transformation logic without requiring manual graph construction. This allows the system to serve itself in creating accurate mappings while improving productivity through automation
2Ease of operation
If high-level abstraction specification is enabled, then ease of operation is improved, but device complexity increases due to automated graph generation
Solution Approach 1:
The system introduces an intermediary automated generation layer that translates high-level user specifications into detailed dataflow graphs. This intermediary component handles the complexity of graph construction while users interact only with simplified schema mapping operations, resolving the contradiction between ease of operation and system complexity
3Productivity
If automated breakdown of problem statements is performed, then productivity is improved, but manufacturing precision may deteriorate due to automatic generation
Solution Approach 1:
The system implements feedback mechanisms where generated dataflow graphs are validated against schema constraints and transformation rules. This feedback loop ensures that automated generation maintains precision by detecting and correcting errors, resolving the contradiction between productivity and accuracy
Data Source
Figure 1A
Figure 1B
Figure 2A~2B
AI summary
Mapping information that specifies attributes of destination entities (308) in terms of attributes of source entities (304-306) is received (402). At least some source entities correspond to respective sets of records in one or more data storage systems (104, 112). The mapping information is processed to generate a procedural specification (120) for computing values corresponding to attributes of destination entities. Collections of nodes (600) are generated (404), each including a first node (602) representing a first relational expression associated with a specified attribute. At least some collections form a directed acyclic graph that includes links to one or more other nodes (604-608) representing respective relational expressions associated with at least one attribute of at least one source entity referenced by a relational expression of a node in the graph. At least two of the collections are merged (406) with each other to form a third collection based on comparing relational expressions of nodes being merged.