Stream Load Shedding Estimation for Database Throughput

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Database systems face performance limitations in processing large amounts of real-time data due to bottlenecks in data processing, which existing technologies fail to address effectively.

Innovation Solution

Implementing a stream-based computing system with load shedding techniques that selectively discard data tuples when processing capacity is overwhelmed, using methods like round robin or importance-based shedding, and estimating the values or positions of discarded tuples to ensure accurate execution of stream operations.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of information

If stream-based computing processes all incoming data tuples, then data processing completeness is improved, but system productivity deteriorates due to bottlenecks

Engineering Contradiction:

Improvedata processing completenessVSAvoidsystem processing throughput

Core Design Contradiction:

Loss of informationVSProductivity

Solution Approach 1:

The patent extracts and discards a portion of incoming data tuples through load shedding mechanisms when processing capacity is exceeded. This selective removal of data elements (tuples) from the processing stream allows the system to maintain operational throughput by preventing complete system shutdown, while still processing the most critical or representative subset of data to preserve essential information integrity.

Inventive Principle:

Principle #2Taking out (Extraction)

2Productivity

If load shedding discards tuples to maintain throughput, then system productivity is improved, but data processing completeness deteriorates

Engineering Contradiction:

Improvesystem processing throughputVSAvoiddata processing completeness

Core Design Contradiction:

ProductivityVSLoss of information

Solution Approach 1:

The patent changes the parameter of data tuple processing by introducing probabilistic load shedding where tuples are discarded based on configurable probability parameters or priority levels. This allows dynamic adjustment of the balance between throughput maintenance and information preservation, enabling the system to adapt to varying load conditions while maintaining operational continuity.

Inventive Principle:

Principle #35Parameter changes

3Measurement precision

If database systems store and index all data before querying, then measurement precision is improved, but response speed deteriorates for real-time applications

Engineering Contradiction:

Improvedata accuracyVSAvoidreal-time processing speed

Core Design Contradiction:

Measurement precisionVSSpeed

Solution Approach 1:

The patent applies preliminary processing actions to incoming data tuples in the stream before they reach the database storage layer. By performing filtering, aggregation, and load shedding operations on the fly during data ingestion, the system prepares and reduces the data set in advance, enabling faster query response times while maintaining data integrity for the processed subset.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS8756335B2Estimating load shed data in streaming database applications

Publication Date: 2014.06.17 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

Techniques are disclosed for processing data streams containing data tuples, where some of the data tuples are load shed. Load shedding refers to a process of selectively discarding some data tuples, e.g., when the amount of received data becomes too overwhelming to manage. When operations are invoked that depend on the values (or count) of the load shed tuples, the value (or counts) of the load shed tuples may be used to perform such operations. For example, an aggregation operation may return the sum, average, mean and/or variance associated with a plurality of tuples processed by a node of the stream application.