Data Persistence for Timely Reports with Batch ETL and Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Business intelligence systems face challenges in generating current reports due to the time-consuming and error-prone manual analysis of real-time data separate from batch data, which is periodically extracted and transformed, leading to reduced performance and outdated insights.
Innovation Solution
A method and system that integrates real-time data acquisition and batch data processing, involving an ETL module for batch data extraction, transformation, and loading, along with a real-time distributed processing module for continuous data indexing and business intelligence report generation, enabling simultaneous analysis of both data types.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If batch data extraction and transformation is performed periodically, then data processing completeness is improved, but data currency and reporting timeliness deteriorate
Solution Approach 1:
The patent segments data processing into two distinct pathways: batch processing for complete historical data analysis and real-time streaming processing for current data insights. This allows the system to maintain both data processing completeness through batch operations and reporting timeliness through real-time operations, resolving the contradiction between these two requirements.
Solution Approach 2:
The patent merges batch processing and real-time processing into a unified architecture where both pathways feed into a common data warehouse and reporting system. This integration allows the system to leverage the completeness of batch data while simultaneously utilizing the timeliness of real-time data for comprehensive business intelligence reporting.
2Speed
If real-time data is collected separately from batch data, then data acquisition speed is improved, but system complexity and manual analysis requirements increase
Solution Approach 1:
The patent combines separate real-time and batch data collection systems into a unified architecture where both data types are ingested through standardized interfaces into a common data warehouse. This merging reduces system complexity by eliminating the need for separate analysis pipelines while maintaining the speed advantage of real-time data acquisition.
Solution Approach 2:
The patent creates a universal data processing platform that handles both real-time and batch data through the same ETL (extract-transform-load) infrastructure. This multi-functional system eliminates the need for separate manual analysis processes, reducing complexity while preserving rapid data acquisition capabilities.
3Loss of time
If batch ETL processing is performed frequently, then data currency is improved, but data source and warehouse performance deteriorate
Solution Approach 1:
The patent segments data processing into real-time streaming operations that continuously update data without heavy transformation, and periodic batch operations that perform comprehensive ETL processing. This segmentation allows frequent data updates for currency while limiting intensive processing to off-peak batch operations, preserving data source and warehouse performance.
Solution Approach 2:
The patent implements periodic batch ETL processing at optimized intervals rather than continuous frequent processing. This periodic action maintains data currency through real-time streaming while allowing batch operations to occur during low-demand periods, preventing performance degradation of data sources and the warehouse.
Data Source
AI summary
A method and system for persisting data are provided. Batch data is periodically extracted via a computer system from at least one primary data source. Batch data is transformed via the computer system. The batch data is loaded, via the computer system, into a database. Real-time data is continuously received from at least one secondary data source. The received real-time data is indexed in an index. Business intelligence reports are generated from both the real-time data in the index and the batch data in the database.