Data Transformation Engine Using Mapping Sheets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Companies face difficulties in quickly and easily converting existing files and databases to new layouts or converting data sources into new databases, often requiring complex rules and additional environments.
Innovation Solution
A transformation engine that uses user-defined mapping sheets to define the conversion of input data sources into output data stores, employing simple rules and metadata in spreadsheets to transform data without needing complex tools or environments, allowing for one-to-one or many-to-many conversions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If large complex rules are used for data conversion, then conversion capability is achieved, but device complexity and ease of operation deteriorate
Solution Approach 1:
The patent divides the complex data conversion rules into separate mapping sheets, each handling a specific transformation aspect. This segmentation allows users to manage complexity by working with isolated, manageable mapping definitions rather than a monolithic complex rule set.
Solution Approach 2:
The patent introduces mapping sheets as intermediary components that mediate between the source data and target data formats. These mapping sheets act as translation layers that simplify the conversion process by providing clear, declarative mappings between different data structures without requiring complex procedural rules.
2Adaptability or versatility
If additional environments are required to run conversion rules, then conversion capability is achieved, but ease of operation deteriorates
Solution Approach 1:
The patent creates a universal transformation engine that can handle multiple data conversion scenarios through a single platform. The engine supports various data types and formats (CSV, JSON, XML, databases) through a unified architecture based on mapping sheets, eliminating the need for separate specialized tools or environments for different conversion tasks.
Solution Approach 2:
The transformation engine is designed to be self-contained, with all necessary conversion logic embedded in the mapping sheets themselves. Users can define transformations directly in the mapping sheets without requiring external processing environments, making the system self-sufficient and easier to operate.
3Manufacturing precision
If complex tools and environments are used for data conversion, then conversion accuracy is achieved, but productivity deteriorates
Solution Approach 1:
The patent replaces complex mechanical processing systems with a more efficient mapping-based approach. Instead of using procedural rules that require interpretation and execution in complex environments, the system uses declarative mappings that directly translate source data to target data, reducing processing overhead and increasing speed while maintaining accuracy.
Data Source
AI summary
The data transformation method operates discrete input data sources DF-in each having df-in data fields carrying to-be-converted data (tbc-data). Initial steps identify tbc-data field and characteristics; output discrete output (df-out) with predefined df-out data fields for converted data (conv-data); and maps or look-up tables for relationships between df-in data fields and df-out data fields. Process activates: SFTP App in the presence df-in and transfers df-in to unstructured data blob; and App copying the df-in from blob to file share archive; first orchestrating sequence App to validate df-in data based upon conversion rules. If INVALID, App generates error entry. If VALID, App converts tbc-data into conv-data with conversion rules and mapping orchestrating App populates df-out data fields with conv-data resulting converted df-out in the blob. A second orchestrating App transfers converted df-out to destination store. Copying App copies converted df-out from blob to archive and deletes converted df-out from blob.