Attribute Definition Mapping for Cross-Schema Data Ingestion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Service providers face challenges in efficiently ingesting data from entities with different schemas, requiring time to modify data storage and retrieval systems when data formats or types change, and managing data from multiple sources with varying definitions.
Innovation Solution
Utilizing an attribute definition file to validate and store data with any schema, allowing seamless ingestion and storage regardless of the data repository's definition, through validation criteria and mapping to known attributes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the service provider uses a fixed data storage and retrieval system with predefined schemas, then data integrity and consistency are maintained, but the system requires time to modify when data formats or types change
Solution Approach 1:
The system separates data ingestion from data storage by introducing an attribute definition file as an intermediate layer. This segmentation allows the storage system to remain fixed while the attribute definition file adapts to new data formats, resolving the contradiction between maintaining data integrity and enabling quick adaptation to format changes.
Solution Approach 2:
The service provider pre-defines attribute definitions in the attribute definition file before data ingestion occurs. This preliminary action establishes the mapping between various data formats and the fixed storage schema in advance, eliminating the need for system modifications when new data formats are received.
2Adaptability or versatility
If the service provider creates a separate data storage system for each entity with different schemas, then each entity's data can be stored according to its specific format, but the overall system complexity increases
Solution Approach 1:
The attribute definition file serves as a universal interface that can define attributes for multiple entities with different schemas. Instead of creating separate storage systems for each entity, the same storage system can handle diverse entity types through the flexible attribute definition mechanism, reducing overall system complexity while maintaining adaptability.
Solution Approach 2:
The attribute definition file acts as an intermediary layer between diverse entity data sources and the unified data storage system. It translates various entity-specific schemas into a common format that the storage system can handle, eliminating the need for multiple specialized storage systems.
3Reliability
If the service provider manually updates data storage systems to accommodate new data types, then data quality control is maintained, but the ingestion process becomes slower and less efficient
Solution Approach 1:
The system enables self-service data ingestion by allowing entities to submit data with new formats without requiring manual system updates. The attribute definition file automatically adapts to new data types through predefined rules and validation logic, maintaining data quality control while significantly improving ingestion efficiency.
Data Source
AI summary
Techniques for ingesting data using an attribute definition file are disclosed. The system receives an attribute definition file that defines an attribute name, an entity type, and one or more validation criteria for an attribute. The system receives a dataset comprising a plurality of attribute values with a respective plurality of attribute names. The system determines whether an attribute name in the dataset maps to an attribute name in the definition file and maps the attribute value in the dataset to the attribute definition of the attribute. The system then determines whether the attribute value meets the validation criteria in the definition and stores the attribute value in a data repository when the value meets the validation criteria.