Compression Rate Prediction Using Federated Process Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Users face challenges in predicting the data compression rate for processes in large systems, making it difficult to balance the benefits of data reduction with potential performance risks, as existing methods lack a quantitative basis for estimating expected compression rates.
Innovation Solution
A system that determines an expected compression rate for prospective processes by compiling and matching compression rate data from existing processes in a federated system using a process name table, with a sliding average that favors more recent data, allowing for prediction without running individual processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data reduction techniques are deployed on a particular process, then the effective capacity utilization is improved, but the performance overhead increases
Solution Approach 1:
The system performs preliminary action by collecting compression rate data from existing processes and storing it in a process name table before the prospective process runs. This pre-collected empirical data allows the system to predict expected compression rates without needing to run the prospective process first, enabling informed decisions about whether to deploy data reduction techniques and thus avoiding unnecessary performance overhead while maintaining effective capacity utilization improvements.
Solution Approach 2:
The system implements feedback by continuously collecting compression rate data from processes that have already executed, storing this empirical information in a process name table, and using it to predict compression rates for prospective processes. This feedback loop enables the system to learn from past performance and make data-driven decisions about deploying data reduction techniques, balancing the benefits of increased capacity utilization against the costs of performance overhead.
2Measurement precision
If compression rate data is collected from all existing processes, then the prediction accuracy is improved, but the system complexity increases
Solution Approach 1:
The process name table serves multiple functions: it stores compression rate data, provides prediction estimates for prospective processes, and can be queried by different host systems in the federated environment. This multi-functional data structure improves prediction accuracy while avoiding the need for separate mechanisms for each function, thus managing system complexity efficiently.
Solution Approach 2:
The system uses copying by storing compression rate data from existing processes in a standardized process name table format. This tabular representation copies essential information (process identifiers, compression rates, timestamps) into a uniform structure that can be efficiently queried and used for predictions, simplifying the system compared to maintaining complex process-specific data structures.
3Quantity of substance
If a federated system is used to share compression data, then the data volume for prediction is improved, but the network overhead increases
Solution Approach 1:
Host systems perform preliminary action by collecting and storing compression rate data locally in the process name table before queries are made. When a prospective process needs prediction, the system queries this pre-populated local data rather than retrieving data from remote hosts in real-time. This reduces network overhead while still providing access to federated system data volume for improved prediction accuracy.
Data Source
AI summary
Determining an expected compression rate for a prospective process in a federated system includes obtaining compression rate data for existing processes in the federated system, compiling the compression rate data into a plurality of entries in a process name table according to process identifier, client, and industry, determining a specific entry in the process name table for an existing process that most closely matches the prospective process, and determining an expected compression rate of the prospective process based on the compression rate data for the specific entry. Compression rate data may be provided by a driver at host systems that sends compression rate information to a central repository. The central repository may be provided by a host system at a data center of the federated system. The compression rate data may use a sliding average that weighs the data more heavily to favor more recent data.