Event-Based Cloud Data Export System
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud-based software products face inefficiencies in exporting and importing large data sets, leading to prolonged downtime and resource utilization due to the need to create compressed backup files from static data, which can take hours and hinder user access to features.
Innovation Solution
Implementing an event-based system that stores and serves event records for data exports and imports, allowing incremental backups by detecting changes since the last export, reducing the data volume to be extracted and downloaded.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional backup methods are used to create compressed backup files from static data, then complete data backup is achieved, but the process takes hours and causes prolonged downtime with reduced user access
Solution Approach 1:
The system performs preliminary actions by continuously capturing and storing data change events in real-time as they occur. Event records are maintained in an event store with timestamps, so when an export is requested, the system only needs to retrieve and process events since the last export rather than recreating the entire backup from static data sources. This preliminary capture of change events dramatically reduces export time while maintaining backup completeness.
Solution Approach 2:
The invention extracts only the necessary portion of data - specifically, only the events that occurred since the last export - rather than processing the entire data set. By querying the event store for events within a specific time range and applying filters, the system extracts minimal data for processing, reducing both export time and resource utilization while ensuring all changes are captured for complete backup.
2Reliability
If traditional backup methods process large data sets, then complete backup coverage is achieved, but resource utilization increases and cost rises
Solution Approach 1:
The system extracts only the essential data changes - events since the last export - rather than processing entire data sets. By querying the event store with time range filters and selecting only relevant event types, the system minimizes the volume of data processed during exports, significantly reducing computational resources, memory usage, and energy consumption while maintaining complete backup coverage.
Solution Approach 2:
The system performs partial action by processing only the necessary subset of data changes rather than the complete data set. By applying filters to event types and limiting queries to specific time ranges, the system processes a partial but sufficient portion of data to achieve complete backup coverage, reducing resource utilization and operational cost.
3Manufacturing precision
If static data is used for exports, then data consistency is ensured, but the export process requires blocking user access causing downtime
Solution Approach 1:
The system performs preliminary action by continuously capturing data changes as events with timestamps before they are needed for export. The event store maintains an immutable record of all changes in chronological order. When an export is requested, the system retrieves pre-captured events since the last export, ensuring data consistency without requiring current data to be static or user access to be blocked.
Solution Approach 2:
The event store acts as an intermediary between the dynamic data sources and the export process. Instead of directly querying live data sources during export (which would require blocking user access), the system queries the event store for changes since the last export. This intermediary layer provides consistent, filtered event data without impacting user access to the underlying data sources.
Data Source
AI summary
Systems and methods for exporting data from a cloud server and/or importing data to a cloud server are disclosed. The method for exporting data includes receiving a plurality of event records. Each event record includes information about an update to application data of an application. The computer-implemented method further including storing the plurality of event records in near real-time in a predefined storage location and in response to a client export request, creating one or more backup files based on the stored event records in the predefined storage location, and allowing export of the one or more backup files to a local client storage location.