Adaptive Chunkfile Archiving for Variable Streaming Data Rates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data platforms face inefficiencies in archiving chunks due to uncertainty in the size of streaming native format data, leading to increased resource consumption and delay when chunks are stored locally or inefficiently sized at archive storage, resulting in higher storage costs and resource usage.
Innovation Solution
A data platform predicts incoming data rates to dynamically determine whether to store chunks locally or directly to archive storage, using machine learning and threshold rates to optimize chunkfile sizing and reduce resource consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If chunks are stored locally before archiving, then chunkfile sizes can be optimized, but memory consumption, bandwidth usage, and processing delay increase
Solution Approach 1:
The patent implements dynamic decision-making by using machine learning models to predict incoming data rates in real-time. The system adapts its archiving strategy based on predicted data rates, dynamically selecting between direct archiving and local buffering modes. This dynamic approach resolves the contradiction by optimizing chunkfile sizes without consistently requiring local storage, thereby reducing memory and bandwidth consumption while maintaining efficiency.
Solution Approach 2:
The system changes the parameter of data rate prediction to determine archiving strategy. By predicting incoming data rates and comparing them against threshold rates, the system adjusts its behavior: when predicted rates exceed thresholds, it buffers locally to optimize chunkfile sizes; when rates are below thresholds, it archives directly to reduce resource consumption. This parameter-based decision-making resolves the contradiction adaptively.
2Quantity of substance
If chunks are archived directly without local storage, then resource consumption is reduced, but chunkfile sizes become inefficient
Solution Approach 1:
The system performs preliminary action by predicting incoming data rates before the actual archiving decision is made. The machine learning model forecasts data rates in advance, allowing the system to proactively choose the optimal archiving path. This preliminary prediction enables efficient chunkfile sizing without consistently requiring local storage, resolving the contradiction by preparing the decision in advance based on predicted conditions.
Solution Approach 2:
The system implements feedback through threshold comparisons. Predicted data rates are fed back against predefined threshold rates to determine the archiving strategy. When predictions exceed thresholds, the system adjusts behavior to buffer locally for optimization; when below thresholds, it archives directly to conserve resources. This feedback mechanism resolves the contradiction by continuously adapting to predicted data rate conditions.
3Manufacturing precision
If local storage is used for all streaming data, then chunkfile optimization is achieved, but processing delay and computing resource usage increase
Solution Approach 1:
The patent applies dynamic decision-making to resolve the time-efficiency contradiction. By using machine learning to predict data rates in real-time and comparing against thresholds, the system dynamically selects between direct archiving (faster, no delay) and local buffering (optimized sizing, potential delay). This dynamic approach ensures chunkfile optimization is applied selectively rather than universally, minimizing processing delay while maintaining efficiency where beneficial.
4Productivity
If direct archiving is used for all data, then processing speed is improved, but storage costs and resource consumption increase
Solution Approach 1:
The system changes the parameter of archiving strategy based on predicted data rates. By predicting incoming data rates and comparing against threshold rates, the system adjusts its behavior: when predicted rates exceed thresholds, it buffers locally to enable efficient chunkfile creation, reducing storage costs; when rates are below thresholds, it archives directly to maximize processing speed. This parameter-based adaptation resolves the contradiction by optimizing for cost-efficiency rather than speed in appropriate scenarios.
Data Source
AI summary
Techniques are described for performing direct archive of data chunkfiles. A computing system comprising a storage device and processing circuitry having access to the storage device may be configured to perform various aspects of the techniques. The processing circuitry may be configured to predict an incoming data rate of native format to be archived to obtain a predicted incoming data rate for the native format data to be archived, and compare the predicted incoming data rate to a first threshold data rate. The processing circuitry may also be configured to, responsive to determining that the predicted incoming data rate exceeds a first threshold data rate, segment the native format data into chunks, and directly write the chunks to an archive storage system as a chunkfile.