Data Aggregation Server Using Pre-Defined Extraction Scripts
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Organizations face challenges in integrating data from multiple operational databases due to differences in data formats and the complexity of custom-designed ETL engines, which are time-consuming and costly to maintain, especially as the organization grows.
Innovation Solution
A system and method for aggregating data from operational databases using pre-defined extraction scripts executed by a data warehouse server, which includes parameters for specifying data to extract, merge, transform, and load data into a multidimensional database, with an interface for modifying scripts to accommodate changing database formats and needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If custom-designed ETL engines are used to integrate data from multiple operational databases, then data integration capability is improved, but system complexity and maintenance cost increase
Solution Approach 1:
The patent introduces a data integration server as an intermediary component that mediates between multiple operational databases and the data warehouse. This server contains pre-defined extraction scripts that serve as standardized intermediaries for data extraction, eliminating the need for custom ETL engine design while maintaining integration capability across diverse database sources
Solution Approach 2:
The data integration server is designed with universal functionality to extract data from multiple different operational database types using a single platform. The pre-defined extraction scripts provide multi-functional capability to handle various database formats and structures without requiring separate custom ETL engines for each database type
2Manufacturing precision
If custom ETL engines are designed for specific organizational needs, then data extraction precision is improved, but development time and cost increase
Solution Approach 1:
The patent applies preliminary action by pre-defining extraction scripts that are prepared in advance for common data extraction scenarios. These scripts are developed beforehand and stored in the data integration server, allowing organizations to immediately extract data with high precision without undergoing lengthy custom development processes
Solution Approach 2:
The pre-defined extraction scripts utilize parameter changes to adapt to different organizational needs. By modifying script parameters rather than rewriting entire ETL engines, the system maintains high data extraction precision while significantly reducing development time and effort
3Productivity
If operational databases are maintained separately by different departments, then operational agility is improved, but data interoperability deteriorates
Solution Approach 1:
The data integration server serves as a mediator that enables interoperability between separately maintained operational databases. It provides standardized extraction interfaces that allow different departmental databases to exchange data effectively, maintaining both operational independence and data interoperability
4Adaptability or versatility
If ETL engine source code is customized to accommodate database format changes, then adaptability to changes is improved, but maintenance complexity increases
Solution Approach 1:
The patent applies parameter changes by enabling adaptation to database format changes through modifying script parameters rather than rewriting source code. The pre-defined extraction scripts contain configurable parameters that can be adjusted to accommodate new database formats, maintaining adaptability while reducing maintenance complexity
Solution Approach 2:
When database formats change, the system can copy existing pre-defined extraction scripts and modify them rather than developing entirely new ETL engines. This copying approach preserves the proven functionality of original scripts while adapting them to new requirements, reducing maintenance burden
Data Source
AI summary
A system for aggregating data from a plurality of operational databases, and a method for providing the same, are provided. The system includes a data store storing a collection of pre-defined extraction scripts. The extraction scripts identify data available for extraction from a plurality of operational database products, and including parameters for specifying which of the data to extract. A data warehouse server executes the extraction scripts to extract, merge, transform and load the specified data from the plurality of operational databases into a multidimensional database.