Transaction Scheduling for Mixed Workloads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data processing systems are inadequate for handling large and complex data sets, as they cannot support mixed workloads on a single copy of data, leading to inefficiencies such as long delays in using recent transactional data for analytical workloads and increased total cost of ownership due to the need for separate processing systems and data duplication.
Innovation Solution
A method and system for scheduling transactions that allow mixed workloads on a single copy of data by enabling short-lived transactions to preempt long-running transactions, thereby supporting prioritized scheduling and reducing the need for separate processing systems and data duplication, using a transaction analyzer, mixing model selector, and transaction execution module to manage transactions and ensure serialization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If transactions are scheduled serially at each executor, then transaction execution simplicity is maintained, but system throughput and responsiveness deteriorate due to inability to support mixed workloads
Solution Approach 1:
The patent segments transactions into different types (short-lived and long-running) and implements separate scheduling queues for each type. The transaction scheduler divides the single serial execution resource into multiple logical queues, allowing short-lived transactions to be processed quickly while long-running transactions proceed separately, thereby improving overall system throughput without requiring complete redesign of the execution engine.
Solution Approach 2:
The patent implements dynamic transaction switching capability where the system can switch between different transaction types based on current system state and workload characteristics. The scheduler dynamically adjusts which queue receives processing attention, enabling the system to adapt to varying workload mixes and optimize throughput for different scenarios.
2Productivity
If long running transactions are allowed to execute, then analytical workload performance is improved, but short-lived transactional transactions are blocked causing delays
Solution Approach 1:
The patent separates transactional and analytical workloads into distinct scheduling queues. Transactional transactions with short lifetimes are placed in one queue while analytical transactions with long running times are placed in another queue. This segmentation allows each queue to be managed independently, ensuring that short-lived transactional transactions are not blocked by long-running analytical transactions.
Solution Approach 2:
The patent ensures continuous processing of both transactional and analytical workloads by maintaining multiple active queues. Instead of allowing one type of transaction to completely block the other, the system continuously processes transactions from both queues according to their respective priorities and characteristics, eliminating idle time and ensuring continuous useful action for both workload types.
3Reliability
If separate systems are used for transactional and analytical workloads, then workload isolation is achieved, but total cost of ownership increases due to data duplication and system maintenance
Solution Approach 1:
The patent merges transactional and analytical workload processing into a single system with unified data storage. Instead of maintaining separate systems with duplicated data, the invention combines both workload types in one system that shares the same data storage infrastructure, reducing data duplication and simplifying system architecture while maintaining workload isolation through logical separation in the scheduling layer.
Solution Approach 2:
The patent creates a universal data system that can handle both transactional and analytical workloads through a single execution engine and shared data storage. The system is designed to be multi-functional, supporting different types of transactions on the same data without requiring separate processing systems, thereby reducing total cost of ownership while maintaining reliability through workload isolation at the scheduling level.
Data Source
AI summary
The present teaching relates to scheduling transactions at a node in a data system. In one example, a request is obtained for performing a first transaction at the node. A mixing condition associated with the first transaction is determined. The first transaction is initiated. The first transaction is suspended when the mixing condition is met. A second transaction is carried out at the node. The first transaction and the second transaction are directed to a same portion of data at the node.