ETL Platform Data Consistency and Rollback
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current analytics systems for businesses lack data consolidation from multiple SaaS applications, provide inconsistent data, and do not support rollback features, leading to data inconsistency and loss of history in case of failures.
Innovation Solution
An ETL platform that fetches consistent datasets in batches, preserves change history by creating new versions instead of overwriting existing data, and allows for rollback to any previous batch, ensuring data consistency and auto-restorability across multiple tenants.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is overwritten with new batches in traditional systems, then storage space is saved, but data history and rollback capability are lost
Solution Approach 1:
The patent segments data storage into versioned batches, where each batch represents a distinct time period's data. Instead of overwriting, new batches are appended with version identifiers, allowing historical data to be preserved while managing storage through selective retention of older batches.
Solution Approach 2:
The system performs preliminary actions by maintaining metadata about all data batches including version information, time periods, and rollback capabilities. This preliminary structuring enables efficient rollback operations without requiring actual duplication of all historical data.
2Adaptability or versatility
If data from multiple SaaS applications is consolidated, then comprehensive analytics are achieved, but data consistency across different sources becomes difficult to maintain
Solution Approach 1:
The patent introduces an intermediary layer (the analytics platform) that sits between multiple SaaS applications and the analytics processing. This intermediary standardizes data ingestion, applies consistent transformation rules, and maintains version control, thereby ensuring data consistency across heterogeneous sources while enabling comprehensive consolidation.
Solution Approach 2:
The system changes parameters by implementing version control metadata and batch tracking mechanisms. Each data batch from different SaaS sources is tagged with version information, time stamps, and source identifiers, allowing the system to track and reconcile data consistency across multiple applications through parameter-based differentiation rather than overwriting.
3Reliability
If rollback capability is added to preserve data history, then data reliability is improved, but system complexity increases
Solution Approach 1:
Instead of maintaining complete redundant copies of all historical data, the patent uses copying selectively through versioned batch metadata and selective data retention. The system copies only essential metadata and selectively retains older batches based on retention policies, reducing the complexity overhead of full rollback capability while maintaining reliability.
4Manufacturing precision
If batch data processing is used to ensure consistency, then data accuracy is improved, but processing time increases
Solution Approach 1:
The patent implements periodic batch processing where data is collected and processed in defined time intervals rather than continuously. This periodic action ensures data consistency within each batch while reducing overall processing time by batching multiple data points together. The system balances accuracy and time by optimizing batch sizes and intervals based on data criticality.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
An extract-transform-load (ETL) platform fetches consistent datasets in a batch for a given period of time and provides the ability to rollback that batch. The batch may be fetched for an interval of time, and the ETL platform may fetch new or changed data from different cloud/on-premise applications. It will store this data in the cloud or on-premise to build data history. As the ETL platform fetches new data, the system will not overwrite existing data, but rather will create new versions so that change history is preserved. For any reason, if businesses would like to rollback data, they could rollback to any previous batch.