Adaptive Data Compression Prediction Engine for Stream Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data compression technologies, such as the Dell Stream Data Platform and Pravega, lack built-in data compression capabilities and require costly compression optimization engines, especially at the client side, where real-time stream management demands high computational performance, and often rely on static compression algorithms that do not adapt to changing data patterns.
Innovation Solution
A predictive and adaptive data compression system that uses a training and inferencing architecture to select an optimal compressor in real-time based on historical data patterns and Service Level Agreement (SLA) constraints, leveraging machine learning models to predict the best compressor for subsequent data batches and adapt to changes in the data stream.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a compression optimization engine is run for each stream batch, then compression performance is improved, but computational cost increases
Solution Approach 1:
The system performs preliminary actions by training a machine learning model offline using historical compression data and performance metrics. The trained model is then deployed to predict optimal compressors for new stream batches, eliminating the need to run expensive optimization engines in real-time while maintaining compression performance.
Solution Approach 2:
The system creates a simplified copy of the compression optimization problem by training a machine learning model that replicates the decision-making process of the full optimization engine. The model learns from historical data and copies the essential patterns, allowing fast predictions without re-running the complete optimization algorithm for each batch.
2Device complexity
If static compression algorithms are used, then device complexity is reduced, but adaptability to changing data patterns deteriorates
Solution Approach 1:
The system transitions from static compression algorithms to a dynamic machine learning-based selector that adapts to changing data patterns. The model is trained on historical data representing various data patterns and compression scenarios, enabling it to dynamically select appropriate compressors based on the characteristics of incoming stream batches while maintaining reasonable system complexity.
Solution Approach 2:
The system changes the parameter of compressor selection from fixed/static to variable/dynamic by using a trained machine learning model that outputs different compressor choices based on input data characteristics. The model learns optimal compressor parameters and selections from historical data, allowing adaptation to changing patterns without requiring complex real-time optimization.
3Speed
If real-time stream management is implemented, then processing speed is improved, but computational resource requirements increase
Solution Approach 1:
The system performs the computationally intensive work of compression optimization in advance by training a machine learning model offline on historical stream data and compression outcomes. Once trained, the model provides fast, low-resource predictions for real-time stream management, achieving both real-time processing speed and reduced computational resource requirements.
Solution Approach 2:
The system substitutes the mechanical/computational process of running optimization engines with a machine learning-based prediction system. The trained model replaces the need for expensive real-time optimization computations, enabling real-time stream management with significantly reduced computational resource requirements while maintaining processing speed.
Data Source
AI summary
Detecting a decrease or loss in performance of a prediction engine configured to predict a compressor for compressing data. The prediction engine suffers a loss in performance when the compressor inferred by the prediction engine does not match or does not sufficiently match a compressor inferred by a compressor selector. Using compressors inferred by two different models allows the loss in performance to be detected. The loss in performance may constitute a violation of a service level agreement (SLA) or service level objective (SLO). Then the loss in performance is determined.