Mixed-Deployment Data Processing Tasks with Cross-Engine Memory Tracking

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

In cluster mixed deployment scenarios, resource contention on nodes, particularly memory contention, leads to significant instability and task failures due to forced process termination by the operating system, impacting the stability and efficiency of big data systems.

Innovation Solution

Implementing a memory tracking mechanism that monitors and manages memory usage across multiple data engines, allowing for adaptive memory allocation and task execution decisions based on the occupancy of computing nodes, thereby preventing memory overuse and ensuring stable operation.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If multiple data engines are deployed on the same electronic device to improve resource utilization, then cluster utilization rate is improved, but memory contention increases causing task failures

Engineering Contradiction:

Improvecluster utilization rateVSAvoidtask execution stability

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The patent implements a memory tracking mechanism that continuously monitors memory usage of multiple data engines and provides feedback to the task scheduling system. This feedback enables dynamic adjustment of task allocation based on current memory occupancy, preventing memory overuse and task failures while maintaining high cluster utilization

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The patent introduces a memory tracking module as an intermediary between multiple data engines and the task scheduling system. This intermediary monitors memory usage across all engines and coordinates task allocation to prevent memory contention, enabling stable task execution in mixed deployment scenarios

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If memory tracking mechanism is implemented to prevent memory overuse, then task execution stability is improved, but system complexity increases

Engineering Contradiction:

Improvetask execution stabilityVSAvoidsystem complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The memory tracking module is designed to serve multiple data engines simultaneously with a single unified implementation. It tracks memory usage across different engine types (OLAP, OLTP, real-time computing) using the same mechanism, reducing overall system complexity while maintaining task execution stability

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS12443440B2Method for executing data processing task in cluster mixed deployment scenario, electronic device and storage medium

Publication Date: 2025.10.14 BEIJING VOLCANO ENGINE TECH CO LTD

AI summary

A method and an apparatus for executing a data processing task in a cluster mixed deployment scenario and a device are provided. A first electronic device is provided with computing nodes of at least two data engines. A specific implementation of the method includes: in response to a first data engine node receiving a first data processing task, determining a first memory occupancy corresponding to the first data processing task, where the first data engine node is deployed on the first electronic device and belongs to a first data engine; and determining whether to execute the first data processing task based on a second memory occupancy corresponding to another data engine node, where the another data engine node is deployed on the first electronic device and belongs to another data engine other than the first data engine in the at least two data engines.