Cost Monitoring for Complex Event Processing Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Complex event processing systems deployed on cloud platforms face unpredictability in resource consumption due to varying load patterns, leading to unexpected and high bills, known as 'bill shock', especially in scalable, distributed streaming systems.
Innovation Solution
A cost monitoring and optimization system that employs a generic cloud-platform independent cost model, multi-query optimization, and operator placement techniques to track and minimize costs in real-time, allowing for near real-time monitoring and optimization of query costs down to the operator level, using a cost calculator component to index operators, assign operator placement algorithms, and deploy optimized queries on scalable streaming systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If automatic scaling is implemented to handle varying load patterns, then system adaptability and productivity are improved, but resource consumption becomes unpredictable and costs increase
Solution Approach 1:
The patent implements cost monitoring that provides feedback to users about their resource consumption and costs in real-time. This feedback mechanism allows users to understand the relationship between their query patterns and resource consumption, enabling them to optimize their queries and control costs while maintaining system adaptability to varying loads.
Solution Approach 2:
The system enables users to self-monitor and self-optimize their query costs through the provided cost monitoring and explanation tools. Users can independently analyze their resource consumption patterns and make adjustments without requiring system-wide changes, thus maintaining adaptability while controlling individual resource usage.
2Measurement precision
If detailed cost monitoring is implemented at operator level, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The patent segments the cost monitoring function into distinct components: cost calculation at the operator level, cost aggregation at the query level, and cost explanation generation. This segmentation allows detailed monitoring without overwhelming system complexity, as each component handles a specific aspect of cost measurement and can be independently optimized.
Solution Approach 2:
The patent introduces cost explanation as an intermediary layer that translates detailed operator-level cost data into meaningful insights for users. This intermediary component aggregates and interprets the detailed measurements, providing precision in measurement while shielding users from the underlying system complexity through simplified cost explanations.
Data Source
AI summary
A cost monitoring system can monitor a cost of queries executing in a complex event processing system, running on top of a pay-as-you-go cloud infrastructure. Certain embodiments may employ a generic, cloud-platform independent cost model, multi-query optimization, cost calculation, and/or operator placement techniques, in order to monitor and explain query cost down to an operator level. Certain embodiments may monitor costs in near real-time, as they are created. Embodiments may function independent of an underlying complex event processing system and the underlying cloud platform. Embodiments can optimize a work plan of the cloud-based system so as to minimize cost for the end user, matching the cost model of the underlying cloud platform.