Streaming Database Ingestion With Per-Table Chunks for Low Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for uploading data into cloud databases suffer from high latency and expense due to manual initiation of data transfer commands, which can lead to data loss and inefficiencies.
Innovation Solution
A streaming ingest platform that organizes data into per-table chunks in per-account blobs, allowing asynchronous upload and immediate query processing before ingestion into target tables, accommodating diverse data sources with varying processing capabilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If manual initiation of data transfer commands is used, then data can be uploaded into the database, but latency increases and data freshness deteriorates
Solution Approach 1:
The system enables self-service data loading where the database automatically receives and processes data without manual command initiation. The streaming ingest platform autonomously manages data transfer, eliminating the need for users to manually trigger copy commands and thereby reducing latency while maintaining automation.
Solution Approach 2:
Data is prepared and staged in advance using the streaming ingest platform before being loaded into the target table. This preliminary action allows data to be ready for immediate ingestion, reducing the time between data generation and availability in the database, thus addressing the latency issue caused by manual initiation.
2Reliability
If manual initiation of data transfer commands is used, then data can be uploaded, but data loss risk increases
Solution Approach 1:
The streaming ingest platform implements feedback mechanisms that monitor data transfer status and provide real-time confirmation of successful data loading. This feedback loop ensures data integrity and prevents loss by verifying that data was successfully transmitted and committed to the database, addressing the reliability concerns associated with manual operations.
Solution Approach 2:
The system employs error handling and retry mechanisms that are pre-configured to cushion against potential data loss. If a data transfer failure occurs, the platform automatically initiates retry operations or falls back to alternative transfer paths, thereby preventing data loss before it can occur during manual data upload operations.
3Ease of manufacture
If running warehouse is used for data transfer, then data can be copied, but operational expenses increase
Solution Approach 1:
The streaming ingest platform extracts and eliminates the need for running warehouse infrastructure from the data transfer process. By using direct streaming ingestion, the system achieves data transfer capability without requiring the expensive running warehouse, thereby reducing operational expenses while maintaining ease of data loading.
Solution Approach 2:
The platform replaces the expensive, persistent running warehouse with a more economical streaming ingest mechanism that uses temporary, disposable computational resources only during data transfer operations. This approach maintains data transfer capability while significantly reducing operational expenses by avoiding the need for continuous warehouse operation.
Data Source
AI summary
A streaming ingest platform can improve latency and expense issues related to uploading data into a cloud data system. The streaming ingest platform can organize the data to be ingested into per-table chunks and per-account blobs. This data may be committed and may be made available for query processing before it is ingested into the target source tables. This significantly improves latency issues. The streaming ingest platform can also accommodate uploading data from various sources with different processing and communication capabilities, such as Internet of Things (IOT) devices.