Dynamic Workload Redistribution in Data Producer-Consumer Systems

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

In consumer-producer distributed systems, existing technologies face challenges in efficiently balancing the workload between database servers and storage devices, leading to suboptimal performance due to inefficient distribution of commutable work.

Innovation Solution

Implementing a workload balancing mechanism where producers and consumers dynamically determine the distribution of commutable work based on their respective workloads, allowing producers to 'push back' unworked data to consumers when they are overloaded, and vice versa, to optimize resource utilization.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If the database server performs all filtering work on data blocks, then the storage device can simply provide raw data, but the database server becomes overloaded with computational tasks

Engineering Contradiction:

Improvestorage device throughputVSAvoiddatabase server computational overload

Core Design Contradiction:

ProductivityVSObject-generated harmful factors

Solution Approach 1:

The system dynamically adjusts the distribution of filtering work between storage device and database server based on real-time workload conditions. The database server can push back filtering tasks to the storage device when it is overloaded, and the storage device can push back to the server when it has capacity, creating a dynamic workload balancing mechanism that adapts to changing system states

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system implements feedback mechanisms where the database server monitors its own workload and the storage device's capacity to perform filtering operations. Based on this feedback, the server makes informed decisions about whether to push back filtering tasks, optimizing the distribution of computational work and preventing server overload while maintaining system performance

Inventive Principle:

Principle #23Feedback

2Productivity

If the storage device performs all filtering work on data blocks, then the database server can focus on query processing, but the storage device becomes a bottleneck

Engineering Contradiction:

Improvedatabase server query processing speedVSAvoidstorage device processing bottleneck

Core Design Contradiction:

ProductivityVSObject-generated harmful factors

Solution Approach 1:

The workload distribution is made dynamic and adaptive. The storage device can push back filtering operations to the database server when it detects high workload or limited capacity, allowing the system to flexibly distribute tasks based on real-time conditions and prevent the storage device from becoming a permanent bottleneck

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The storage device monitors its own workload and capacity through feedback mechanisms, enabling it to make informed decisions about whether to perform filtering operations or push back to the database server. This feedback loop ensures optimal workload distribution and prevents the storage device from becoming overloaded

Inventive Principle:

Principle #23Feedback

3Device complexity

If fixed workload distribution is used between producer and consumer, then system configuration is simple, but system performance cannot adapt to varying workloads

Engineering Contradiction:

Improveworkload distribution configurationVSAvoidsystem performance adaptability

Core Design Contradiction:

Device complexityVSProductivity

Solution Approach 1:

The system transitions from static to dynamic workload distribution, where the database server and storage device continuously adjust the division of filtering tasks based on real-time workload conditions. This dynamic approach maintains simplicity in configuration while significantly improving performance adaptability to varying system demands

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system implements self-service workload balancing where the database server autonomously monitors its own workload and the storage device's capacity, making informed decisions about task distribution without external intervention. This self-service mechanism simplifies configuration while enabling adaptive performance optimization

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS8935227B2Redistributing computation work between data producers and data consumers

Publication Date: 2015.01.13 ORACLE INT CORP

AI summary

In a consumer-producer distributed model of computing, consumers request data from producers and perform computing work on the data received from the producers, thereby transforming the data. At least some of the computing work performed by a consumer on data provided by the producer is commutable between the consumer and producer, that is, either the producer as well as the consumer can perform the computing work on the data. A producer redistributes commutable based on the work load of the producer and consumer as well as other factors.