Schema Inheritance for Multi-Source Data Transformation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data transformation techniques face challenges in handling data from multiple sources with different formats, requiring manual programming for each format and struggling with data normalization and redundancy, leading to inefficient storage and retrieval.
Innovation Solution
Employing Object-Oriented Programming (OOP) principles to create a base schema with template metadata, deriving stage schemas from it, and using inheritance to dynamically update a master schema, enabling efficient data aggregation and transformation across diverse data sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If manual programming is used for each data format, then data transformation can be performed, but the device complexity and programming effort increase significantly
Solution Approach 1:
The patent creates a universal base schema that can handle multiple data formats through inheritance. The base schema contains common metadata and transformation logic that is reused across all data sources, eliminating the need for separate programming for each format while maintaining adaptability to different data structures
Solution Approach 2:
The patent segments the data transformation system into hierarchical components: base schema (common functionality), stage schemas (intermediate processing), and data source-specific schemas. This segmentation allows complex transformation logic to be divided into manageable, reusable units that reduce overall programming complexity
2Productivity
If data from multiple sources with different formats is stored without normalization, then storage is simpler, but data redundancy increases and retrieval efficiency decreases
Solution Approach 1:
The patent performs data normalization in advance during the staging process. Data is transformed into a standardized format and stored in stage schemas before final retrieval, eliminating the need for complex normalization queries at retrieval time and improving overall productivity
Solution Approach 2:
The patent introduces stage schemas as intermediary structures between raw data sources and the final master schema. These stage schemas serve as intermediate storage that pre-processes and normalizes data, reducing redundancy before data reaches its final destination and improving retrieval efficiency
3Productivity
If separate processing is used for each data source, then data format specificity is maintained, but the loss of time for data aggregation increases
Solution Approach 1:
The patent merges multiple data sources into a unified master schema through inheritance. All data sources inherit common transformation logic from the base schema, allowing parallel processing of multiple sources while maintaining consistency and reducing overall aggregation time through shared functionality
Data Source
AI summary
Systems and methods for transforming data using database are provided. In examples, a method includes instantiating a base schema having template metadata and deriving a plurality of stage schemas from the base schema. Each stage schema is associated with a respective data source, includes a respective stage table populated from the respective data source, and includes metadata inherited from the template metadata when the respective stage schema was derived from the base schema. The method further includes receiving a plurality of sets of raw data being associated with respective data sources of the plurality of data sources. The method further includes extracting one or more respective data items from each stage table using a respective mapping from the corresponding stage schema to a master schema.