Orchestrator Compressibility Settings for Data Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In shared computing environments, storage nodes face challenges in differentiating between data streams from various applications, leading to unnecessary compression attempts that waste computational resources, as they cannot accurately determine the compressibility of data from different sources.
Innovation Solution
An orchestrator system assigns compressibility settings to data streams based on previous compression ratio measurements, using application identifiers to differentiate between data streams and avoid compressing already compressed data, thereby optimizing storage and computation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If storage nodes attempt to compress all data streams, then storage space savings may be achieved, but computational resources are wasted on already compressed data
Solution Approach 1:
The system performs preliminary compression attempts on data streams before final storage, measuring compression ratios in advance. This preliminary action identifies which data streams are already compressed (showing poor compression ratios) versus those that can benefit from compression, allowing the system to make informed decisions about subsequent storage actions and avoid wasting computational resources on already compressed data.
Solution Approach 2:
The system implements a feedback mechanism where compression ratio measurements from previous compression attempts are fed back into the system. This feedback allows the orchestrator to learn from past compression results and adjust future compression decisions, identifying patterns in data compressibility across different applications and data streams to optimize resource allocation.
2Quantity of substance
If storage nodes compress all incoming data, then storage efficiency improves, but system complexity increases due to inability to differentiate data stream types
Solution Approach 1:
The system segments data streams by application identifier, treating each application's data stream as a distinct entity. This segmentation allows the orchestrator to apply different compression strategies to different applications based on their individual compressibility characteristics, rather than applying a uniform compression approach to all data, thereby reducing unnecessary computational overhead.
Solution Approach 2:
The system changes the compression parameter (whether to compress) based on measured compression ratios for each application. By dynamically adjusting this parameter according to empirical data rather than using a fixed rule, the system simplifies decision-making logic while improving storage efficiency through targeted compression of only those data streams that benefit from it.
3Quantity of substance
If compression is attempted on all data streams, then potential storage savings are achieved, but processing time increases
Solution Approach 1:
The system applies compression partially - only to data streams that demonstrate compressibility through measured compression ratios. Rather than applying compression universally (excessive action), the system selectively compresses only those data streams where compression provides actual benefit, thereby reducing total processing time while maintaining storage savings.
Data Source
AI summary
Systems, methods, and apparatus are disclosed for management of data storage with stream based compressibility. In an example, an orchestrator including a storage controller managing one or more storage nodes executes on one or more processors. The orchestrator receives compression ratio measurements associated with attempts to compress data from a plurality of applications running on a plurality of service guests. The orchestrator receives a request for storing a first data stream from a first application running in a first service guest. Based on a compression ratio of previously stored data associated with the first application being less than a threshold, the orchestrator assigns a first compressibility setting to the first data stream. The first compressibility setting causes a storage node to store the first data stream without attempting to compress the first data stream.