Dynamic Data Log Ingestion Throttling via Age Threshold
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data storage systems face limitations in ingesting incoming data due to finite data logs and resource bottlenecks, leading to abrupt variations in data ingestion speeds, which negatively impact host application performance and user experience.
Innovation Solution
A technique that dynamically controls the acceptance of host application data into a data log by comparing the age of the oldest entry with a dynamically updated age threshold, temporarily refusing new entries when the oldest entry exceeds the threshold and placing them in a pending list, thereby preventing resource overload.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the data log accepts all incoming data without throttling, then data ingestion speed is high, but the data log may become full and cause IO requests to be refused
Solution Approach 1:
The system continuously monitors the age of the oldest log entry and compares it against a dynamically updated age threshold. This feedback mechanism allows the data log to adjust its acceptance behavior in real-time, preventing fullness while maintaining high ingestion speeds when resources are available.
Solution Approach 2:
The age threshold is dynamically updated based on system heuristics rather than being fixed. This dynamic adjustment allows the system to adapt to changing load conditions, optimizing the balance between accepting new data and preventing resource exhaustion.
2Reliability
If the data log refuses new entries when the oldest entry is old, then resource overload is prevented, but data ingestion speed decreases
Solution Approach 1:
The system takes preliminary action by refusing new log entries before the data log becomes full. By monitoring the age of the oldest entry and comparing it to the threshold, the system prevents resource overload conditions from developing, maintaining system stability proactively rather than reactively.
Solution Approach 2:
The system prepares by maintaining a pending list of IO requests that cannot be immediately processed. This preliminary organization of refused requests allows for efficient later processing when resources become available, minimizing the impact on overall productivity.
3Reliability
If host applications retry IO requests after data log refusal, then data can eventually be written, but application performance becomes inefficient due to abrupt speed changes
Solution Approach 1:
The system preliminarily stores refused IO requests in a pending list rather than immediately rejecting them. This preliminary action preserves the requests in an organized manner, allowing the host application to continue processing without interruption while the data log prepares to accept them later.
Solution Approach 2:
The pending list acts as an intermediary between the host application and the data log. It buffers the abrupt transitions in data ingestion speed by holding requests temporarily, smoothing out the performance impact on host applications while ensuring data eventually reaches the log.
Data Source
AI summary
A technique for controlling acceptance of host application data into a data log in a data storage system includes selectively accepting or refusing newly arriving host data into the data log based on a comparison between an oldest entry in the data log and an age threshold. The age threshold is dynamically updated based on system heuristics. As long as the oldest log entry is younger than the age threshold, the data log continues to accept newly arriving host application data, acknowledging IO requests to host applications as the data specified in those requests is entered into the log. However, when the oldest log entry is older than the age threshold, new log entries are temporarily refused entry into the data log. Instead, newly arriving data are placed in a pending list, where they are kept until the data log is again accepting new log entries.