Hierarchical Data Normalization for Heterogeneous Sources
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage services face challenges in standardizing and organizing heterogeneous user data from various sources, which are stored using different mapping and formats, making it difficult to present and access this data in a coherent manner.
Innovation Solution
The system ingests heterogeneous data objects and converts them into standardized formats, associating them with profiles and maintaining a hierarchy within these profiles, using object mapping and key indexing to ensure data objects can be searched and accessed efficiently across different sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If heterogeneous data from various sources is stored using different mapping and formats, then data can be preserved in its original form from different sources, but it becomes difficult to present and access this data in a standardized hierarchical way
Solution Approach 1:
The patent introduces a data normalization service as an intermediary between heterogeneous data sources and the data store. This service converts data from various formats (JSON, XML, CSV, etc.) into a standardized hierarchical format using configuration files that define the expected structure. The intermediary layer preserves the ability to accept diverse input formats while providing a unified access interface, resolving the contradiction between adaptability and ease of operation.
Solution Approach 2:
The system segments the data storage architecture into distinct layers: raw data ingestion, normalization/standardization, and hierarchical organization. By dividing the data processing pipeline into separate stages with clear responsibilities, the system can handle heterogeneous data formats at the ingestion layer while presenting a standardized view at the access layer, thus maintaining both format compatibility and access simplicity.
2Manufacturing precision
If data objects are ingested and normalized from heterogeneous sources, then data standardization is achieved, but the complexity of handling multiple formats and mappings increases
Solution Approach 1:
The patent changes the parameter of data representation from heterogeneous formats to a standardized hierarchical format. By defining a target data model with specific structure requirements and using configuration files to map source formats to this model, the system achieves data standardization while managing complexity through parameter transformation rather than complex processing logic.
Solution Approach 2:
The system performs preliminary normalization and validation of data objects during the ingestion phase, before they are stored or accessed. Configuration files pre-define the expected data structure and mappings, allowing the normalization service to automatically transform and validate data without requiring complex runtime processing, thus achieving standardization while controlling complexity.
3Stability of the object's composition
If a hierarchical structure is imposed on normalized data objects, then coherent organization is achieved, but the processing and mapping operations become more complex
Solution Approach 1:
The patent creates a universal hierarchical data model that can accommodate multiple types of data objects from different sources. The standardized format serves as a multi-functional container that can represent various data structures (contacts, accounts, orders, etc.) in a unified way. This universal model simplifies mapping operations by providing consistent structure rules that apply across all data types, achieving coherent organization without proportionally increasing mapping complexity.
Data Source
AI summary
Systems and methods are described herein relating to ingesting data objects from various sources and associating them in a hierarchy with a profile. In some aspects, a data object, including a set of fields and indicating a data object type, may be obtained for associating with a profile maintained by a computing resource service provider. The data object may be converted into a standardized format based on the data object type indicated in the data object using an object mapping associated with the data object. A key associated with the data object and indicated in the object mapping may be selected and used to search an index for a related profile. Upon finding a profile with the matching key, the profile may be updated to include an association of the standardized data object, such as in a hierarchy.