Schema-Mapped Data Ingestion for Reusable CSV Data Cleansing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Entities face challenges in efficiently ingesting and processing data due to differing data formats and structures, leading to significant labor, time, and cost expenditures, especially when interfacing with multiple partner entities.
Innovation Solution
A data-ingestion service that translates structured data files, such as CSV files, to an internal schema using a mapping library, with user interfaces for manual mapping of unmapped fields and validation, and learns from user inputs to improve future data processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual inspection and verification of data is performed, then data accuracy is improved, but time consumption and labor costs increase
Solution Approach 1:
The system performs self-service by automatically detecting data format discrepancies and performing translations without requiring manual inspection. The translation service autonomously identifies schema differences between entities and applies appropriate translation rules, eliminating the need for human verification while maintaining accuracy.
Solution Approach 2:
The patent replaces the mechanical manual verification process with an automated computational system. The translation service uses algorithmic methods to detect schema differences and apply translations, substituting human labor with automated processing that maintains accuracy while dramatically reducing time consumption.
2Manufacturing precision
If custom scripts are written to parse data files, then data processing accuracy is improved, but development costs and maintenance complexity increase
Solution Approach 1:
The translation service provides a universal solution that handles multiple data formats and schemas through a single platform. Instead of requiring separate custom scripts for each entity or data type, the service universally applies translation rules across diverse data sources, reducing maintenance complexity while maintaining processing accuracy.
Solution Approach 2:
The patent introduces a translation service as an intermediary layer between data sources and the target system. This intermediary automatically handles schema translation and format conversion, eliminating the need for complex custom parsing scripts while maintaining data processing accuracy through standardized translation mechanisms.
3Productivity
If data format standardization is implemented across all entities, then processing efficiency is improved, but adaptability to different entity formats decreases
Solution Approach 1:
The translation service implements dynamic adaptability by automatically detecting the source entity's data format and schema structure, then applying appropriate translation rules tailored to that specific format. This dynamic approach maintains processing efficiency while preserving adaptability to different entity formats, as the system adjusts its translation strategy based on the incoming data characteristics.
Solution Approach 2:
The patent changes the parameter of format handling from static standardization to dynamic translation. Instead of requiring all entities to adopt a single standardized format, the system translates various formats into a compatible internal schema, maintaining processing efficiency while preserving adaptability to different source formats through parameter-based translation rules.
4Reliability
If recurring data exchanges are performed to account for data changes, then data currency is improved, but time and resource costs increase
Solution Approach 1:
The translation service performs preliminary action by establishing translation rules and schemas in advance for each entity. When data changes occur, the pre-configured translation framework is already in place, allowing for rapid translation without requiring recurring manual configuration or extensive processing time, thus maintaining data currency while reducing exchange overhead.
Data Source
AI summary
A system and method for ingesting data is disclosed. In example embodiments, the method may include receiving a structured data file, such as a comma separated value (CSV) file, and translating the file to an internal schema. Translating the file to the internal schema may including mapping one or more incoming fields of the file to one or more internal fields of the internal schema. The method may further include displaying a user interface and receiving, via the user interface, a mapping of an unmapped incoming field to a selected internal field. The method may further include receiving a second structured data file, which may include a second plurality of incoming fields, including the unmapped field. The method may further include mapping the unmapped field of the second file to the selected internal field.