AI Data Processing System for Warehouse Query Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data analysis in data warehouses often requires complex queries to access relevant data, leading to inefficiencies and increased costs, while existing data processing systems lack the ability to ensure data quality and optimize resource allocation for data uploads.
Innovation Solution
An AI-based data processing system that preprocesses data to identify anomalies and errors, uses machine learning models to generate predictions, and dynamically determines resource requirements for data uploads, ensuring data quality and optimizing resource allocation through incremental loading and model retraining.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If complex queries are performed in data warehouses to access relevant data, then data accessibility is achieved, but processing time and computational resources increase
Solution Approach 1:
The patent divides the data warehouse into multiple data marts, each serving specific business domains (sales, manufacturing, etc.). This segmentation allows users to query only relevant data from appropriate data marts rather than performing complex queries across the entire data warehouse, thereby reducing processing time while maintaining data accessibility.
Solution Approach 2:
The system performs preliminary data extraction and transformation into data marts before actual query operations. By pre-processing and organizing data into subject-oriented structures aligned with business units, the system enables faster query execution without requiring complex real-time processing during data access operations.
2Reliability
If data is transferred from source databases to destination databases using ETL process, then data integration is achieved, but processing time and computational resources increase
Solution Approach 1:
The ETL process is segmented into multiple stages: data extraction from source databases, transformation into standardized formats, validation for quality assurance, and loading into destination data marts. This segmentation allows each stage to be optimized independently and enables parallel processing, reducing overall ETL time while maintaining integration quality.
Solution Approach 2:
The system performs preliminary data extraction and transformation into data marts before actual query operations. By pre-processing and organizing data into subject-oriented structures aligned with business units, the system enables faster query execution without requiring complex real-time processing during data access operations.
3Reliability
If data quality checks are performed to identify anomalies, then data reliability is improved, but processing time increases
Solution Approach 1:
The system performs data quality checks and anomaly detection as preliminary actions during the ETL process, before data is made available for querying. By validating data integrity, checking for anomalies, and ensuring quality standards are met during the transformation stage, the system guarantees data reliability without requiring separate time-consuming verification steps later in the workflow.
Data Source
AI summary
An Artificial Intelligence (AI)-based data processing system processes current data to determine if the quality of the current data is adequate to be provided to data consumers and if the quality is adequate, the current data is further analyzed to determine if an impacted load including changes to dimension data of the current data or an incremental load including changes to fact data of the current data is to be provided to the data consumers. Depending on the amount of data to be provided to the data consumers, processing units (PUs) may be determined and assigned to carry out the data upload. Various machine learning (ML) models that are used to provide predictions from the current data are analyzed to determine the quality of predictions and if needed, can be automatically retrained by the data processing system.