Decoupled OLAP Database Creation via Preliminary Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional data processing methods for creating and updating OLAP databases from OLTP systems are inefficient, relying on relational models that are slow and resource-intensive, requiring extensive IT infrastructure and data scientist dependency, and lacking agility in data exploration and insights generation.
Innovation Solution
The implementation of a decoupled extract, load, and transform (ELT) process that extracts data from OLTP systems in advance, using a non-relational key-value pair approach with in-memory columnar stores, applying transformations, and storing data in optimized OLAP databases, allowing for incremental updates and efficient querying without redundant data access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional relational database methods are used to create and update OLAP databases from OLTP systems, then data accuracy and completeness are maintained, but the process is slow and resource-intensive with high computational latency
Solution Approach 1:
The system performs preliminary extraction of data from OLTP systems in advance, storing it in bulk data stores before it is needed for OLAP database creation. This advance preparation eliminates the need for time-consuming data extraction during the OLAP database creation process, significantly reducing computational latency and enabling faster database creation and updates.
Solution Approach 2:
The invention extracts the data extraction function from the OLAP database creation process, separating it into a preliminary step that occurs independently. By taking out the extraction operation and performing it in advance, the system eliminates redundant data access during the actual OLAP database creation, improving speed while maintaining data accuracy.
2Manufacturing precision
If extensive data cleaning and processing is performed to ensure data quality, then data accuracy is improved, but the process becomes more complex and resource-intensive
Solution Approach 1:
Data cleaning and validation operations are performed in advance during the preliminary extraction phase, before data is loaded into bulk data stores. This preliminary processing ensures data quality is established early, reducing the need for complex processing operations during OLAP database creation and simplifying the overall data processing pipeline.
3Adaptability or versatility
If multiple separate extraction processes are used for different OLAP databases, then each database can be customized, but the overall process becomes inefficient with redundant data access
Solution Approach 1:
The system merges multiple separate extraction processes into a single unified preliminary extraction process that extracts data for multiple OLAP databases simultaneously. This consolidated approach eliminates redundant data access operations while maintaining the ability to customize each OLAP database through selective data loading and transformation, thereby improving overall productivity without sacrificing adaptability.
Solution Approach 2:
The preliminary extraction process is designed to be universal, capable of extracting data for multiple different OLAP databases in a single operation. This multi-functional extraction process serves all OLAP database creation needs, eliminating the inefficiency of separate extraction processes while preserving customization capabilities through configurable data selection and transformation rules.
Data Source
AI summary
The technology disclosed relates to creating and frequently updating multiple online analytic processing (OLAP) analytic databases from an online transaction processing (OLTP) transaction updatable system that includes transaction commit, rollback, and field level security capabilities. It also relates to transparently decoupling extraction from rebuilding of frequently updated OLAP analytic databases from the OLTP transaction updatable system.