Master Data Ingestion via Analytics Schema Conversion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The integration of master data from multiple applications with Internet of Things (IOT) devices is challenging due to differing schemas and the need for efficient processing and analysis, particularly in handling massive data volumes and semi-structured formats.
Innovation Solution
A method is provided for ingesting and processing master data by converting it into an analytics schema, using mappings to standardize different schemas, and annotating IOT data with metadata to facilitate storage and processing, allowing for efficient combination with time series data and efficient data processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If master data from multiple applications with different schemas is integrated directly, then data integration capability is improved, but system complexity increases due to schema differences and processing requirements
Solution Approach 1:
The patent introduces an intermediary component (data integration service or schema mapping layer) that sits between multiple applications with different schemas and the target system. This intermediary automatically transforms and adapts data from various source schemas to a unified target schema, enabling data integration without directly coupling the heterogeneous systems. The intermediary handles schema differences, data type conversions, and structural reconciliations, thereby improving adaptability while containing system complexity within the intermediary layer.
Solution Approach 2:
The patent employs parameter changes by dynamically adjusting schema mapping parameters and data transformation rules based on the source application and target requirements. The system maintains configurable schema mappings that can be modified to accommodate different data sources, allowing the integration mechanism to adapt to varying schemas through parameter adjustments rather than structural changes. This enables flexible data integration while keeping the core system architecture stable and manageable.
2Productivity
If master data is converted to analytics schema with additional data retrieval, then data analysis capability is improved, but processing time increases
Solution Approach 1:
The patent applies preliminary action by pre-converting master data to the analytics schema and pre-retrieving additional required data elements before actual analysis operations begin. The system performs schema transformations and data enrichment in advance, so that when analysis queries are executed, the data is already in the optimal format. This upfront preparation reduces the processing time during actual analysis operations, balancing improved data analysis capability with acceptable processing time.
Solution Approach 2:
The patent implements continuity of useful action by maintaining persistent connections and cached mappings between source schemas and analytics schemas. Once schema mappings are established and additional data elements are identified, the system maintains these relationships and reuse them for subsequent data conversions. This continuous state avoids repeated schema resolution and data retrieval operations, reducing processing time while sustaining high data analysis capability across multiple operations.
3Loss of information
If additional data elements are retrieved based on message contents, then data completeness is improved, but resource consumption increases
Solution Approach 1:
The patent applies partial action by selectively retrieving only the specific additional data elements that are actually needed based on the message contents and analysis requirements. Rather than fetching all possible additional data, the system identifies and retrieves only the necessary subset of data elements required to complete the analysis. This selective approach ensures data completeness for the specific analysis task while minimizing resource consumption by avoiding unnecessary data retrieval and processing of extraneous information.
Data Source
AI summary
Techniques and solutions are provided for integrating master data from multiple applications. Master data from multiple applications can be integrated for use in processing data associated with internet of things (IOT) devices, such as by joining master data with timeseries data (including aggregated values). Integrating master data from multiple applications can include converting master data from a schema used by an application into an analytics schema. New or updated master data can be indicated in a message sent by an application. In processing the message, additional master data, or data used to determine how master data should be processed, can be retrieved.