OLAP Network Data Structure Eliminates ETL Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing OLAP systems require significant effort to adapt to data structures of OLTP systems, especially when data is remotely linked via multiple intermediate objects, and data updating is not fast enough, leading to time-consuming ETL procedures that hinder real-time analysis.
Innovation Solution
Implementing an OLAP system with a network data structure that stores transactional data in an in-memory database, allowing for real-time data updates and easier report generation without the need for periodic ETL procedures, using a column-oriented database and network data structure that relates data objects with relationships, enabling faster data access and reporting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If traditional ETL procedures are used to transfer data from OLTP to OLAP systems, then data can be updated periodically, but the process is time-consuming and prevents real-time analysis
Solution Approach 1:
The patent extracts the ETL transformation logic into a separate mapping layer that sits between OLTP and OLAP systems. This mapping layer uses XML-based metadata to define relationships between source and target data structures, allowing data to be transferred and transformed without requiring complex ETL procedures. The extraction separates the data movement function from the analytical processing function, enabling real-time data availability in OLAP systems.
Solution Approach 2:
The patent performs preliminary actions by pre-defining data mappings and relationships in an XML metadata repository before data transfer occurs. The mapping layer contains pre-configured transformation rules that automatically apply during data insertion, eliminating the need for time-consuming ETL transformation steps at data transfer time. This preliminary configuration enables immediate real-time analysis capability.
2Ease of manufacture
If star schema is used in OLAP systems, then data can be organized for analysis, but modeling effort and complexity increase significantly
Solution Approach 1:
The patent creates a universal mapping layer that can handle multiple data sources and target structures through a single XML-based metadata framework. This mapping layer serves multiple functions: data transformation, relationship definition, and query optimization. By making the mapping layer universal and configurable through metadata, the system reduces the complexity of implementing star schemas while maintaining ease of report generation.
Solution Approach 2:
The patent introduces an intermediary mapping layer between OLTP and OLAP systems that mediates the complexity of data structure transformations. This mapping layer uses XML metadata to define relationships without requiring direct modification of star schema structures. The intermediary handles the complexity of data mapping, allowing users to generate reports easily while the system manages the underlying structural complexity.
3Adaptability or versatility
If data is remotely linked via multiple intermediate objects in OLTP systems, then data relationships can be represented, but adapting to this structure in OLAP systems requires significant effort
Solution Approach 1:
The patent implements dynamic adaptability through XML-based metadata that can be configured to represent various data relationship structures without requiring system redesign. The mapping layer dynamically adapts to different OLTP data structures by loading appropriate mapping definitions from XML files, allowing the OLAP system to work with remotely linked data through multiple intermediate objects without increasing modeling effort.
Data Source
AI summary
In one embodiment the present invention includes a computer implemented method of relating data and generating reports. The method includes storing, by an OLAP system, a network data structure that relates a plurality of data objects. The method further includes storing transactional data in an in-memory database in the OLAP system. The method further includes generating, by the OLAP system, a report using the stored transactional data according to the network data structure. In this manner, deficiencies of the traditional star schema paradigm of data warehousing may be avoided.