Adaptive Data Ingestion Rates for Cluster Computing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data ingestion methods in cluster-computing frameworks, such as Apache Spark, are inefficient for smaller data input sizes and fail to adapt to varying data flow rates, leading to suboptimal resource utilization in data lakes.
Innovation Solution
The method determines a moving average streaming rate for each data stream and adjusts the ingestion frequency based on a platform-preferred ingestion rate and data flow rates, using techniques like exponentially weighted moving averages to optimize data ingestion and resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data ingestion uses fixed rate methods, then the system is simple to operate, but it cannot adapt to varying data flow rates and leads to suboptimal resource utilization
Solution Approach 1:
The patent implements dynamic ingestion rates by calculating moving averages of data flow rates and adjusting ingestion frequencies accordingly. The system transitions from static fixed-rate ingestion to dynamic adaptive ingestion where the ingestion rate automatically adjusts based on actual data stream characteristics, resolving the contradiction between adaptability and complexity through automated dynamic adjustment.
Solution Approach 2:
The system employs feedback mechanisms by continuously monitoring actual data flow rates and using this information to adjust ingestion frequencies. The moving average calculation provides feedback about actual data arrival patterns, which then feeds back into the ingestion rate determination, creating a closed-loop system that adapts to varying data flow rates while managing complexity through systematic feedback processing.
2Productivity
If data is ingested at high rates continuously, then data processing throughput is improved, but resource utilization becomes inefficient and processing time increases for smaller data inputs
Solution Approach 1:
The patent changes the parameter of ingestion frequency based on calculated moving average data flow rates. By adjusting the ingestion frequency parameter dynamically - increasing it when data flow rates are high and decreasing it when data flow rates are low - the system optimizes both throughput and resource utilization efficiency, resolving the contradiction between productivity and energy loss.
Solution Approach 2:
The system applies partial action by ingesting data at frequencies that match actual data availability rather than continuously at maximum capacity. This prevents excessive action (ingesting more data than available) and optimizes resource usage by aligning ingestion intensity with actual data flow conditions, thereby reducing wasted computational resources.
3Loss of time
If the system processes smaller data input sizes, then processing time is reduced, but existing ingestion methods become inefficient
Solution Approach 1:
The dynamic ingestion rate adjustment directly benefits smaller data inputs by reducing the time spent waiting for data to accumulate. The system calculates moving averages and adjusts ingestion frequencies in real-time, allowing efficient processing of smaller data batches without the inefficiencies of fixed-rate methods, thus resolving the contradiction between processing time and productivity.
Data Source
AI summary
Described are techniques for data ingestion including determining a respective moving average streaming rate for each of a plurality of incoming data streams to a cluster-computing framework. The techniques further include determining a respective ingestion frequency for each of the plurality of incoming data streams by dividing a platform-preferred ingestion rate of the cluster-computing framework by a respective moving average streaming rate of each of the plurality of incoming data streams. The techniques further include ingesting each of the plurality of incoming data streams to the cluster-computing framework at the respective ingestion frequency.