Distributed Computing Framework Operator Scope Description

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing distributed computing frameworks, such as Apache Spark and Google CloudDataflow, cannot describe the scope of data an operator acts on or handle nested grouping operations, limiting their ability to construct logical plans for complex distributed computing tasks.

Innovation Solution

A task processing method and distributed computing framework that parses expressions to construct task description information, describing the relationship between operators and distributed datasets, allowing operators to act on grouped datasets, and performing distributed computing based on this information to handle complex task requirements.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If distributed computing frameworks use traditional logical plans to describe upstream and downstream relationships of operators, then the framework structure is simple, but the framework cannot describe the scope of data that the operator acts on or handle nested grouping operations

Engineering Contradiction:

Improvecapability to describe operator acting scope and nested grouping operationsVSAvoidlogical plan construction complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent segments the logical plan into multiple levels: global level (overall distributed dataset) and local level (grouped datasets). This segmentation allows operators to be associated with specific data scopes through domain nodes, enabling description of nested grouping operations while maintaining clear structural organization.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent implements a nested domain structure where local domains (grouped datasets) are contained within global domains (distributed datasets). This nested structure allows operators to act on both global and local datasets simultaneously, describing complex nested grouping operations like grouping data and then operating on the grouped data.

Inventive Principle:

Principle #7Nested doll (Nesting)

2Adaptability or versatility

If the framework supports only global distributed datasets, then the framework is easy to operate, but the framework cannot handle operations requiring grouped data to be grouped again

Engineering Contradiction:

Improvesupport for nested grouping operationsVSAvoidoperational simplicity

Core Design Contradiction:

Adaptability or versatilityVSEase of operation

Solution Approach 1:

The patent introduces dynamic domain switching capability where operators can be configured to act on either global or local datasets based on their domain specifications. This dynamic behavior allows the framework to adapt to different operation types (global operations vs. nested grouping operations) while maintaining a unified operational interface.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The patent introduces domain nodes as intermediaries between operators and datasets. These domain nodes mediate the relationship by specifying whether an operator acts on global or local data, enabling nested grouping operations without complicating the operational interface for end users.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Loss of information

If the framework uses traditional operator relationship description, then the framework structure is simple, but the constructed logical plan cannot describe requirements such as grouping data and then operating on the grouped data

Engineering Contradiction:

Improveinformation about operator acting scope and data groupingVSAvoidlogical plan structure

Core Design Contradiction:

Loss of informationVSDevice complexity

Solution Approach 1:

The patent applies local quality by allowing different parts of the logical plan to have different characteristics. Global domain nodes describe overall data scope while local domain nodes describe grouped data scope. Operators can be associated with specific domain levels, preserving information about their acting scope without requiring a completely complex logical plan structure.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The patent adds a domain dimension to the traditional operator relationship description. Instead of only describing upstream-downstream relationships, the logical plan now includes domain level information (global vs. local), creating a multi-dimensional description that captures operator acting scope and nested grouping operations.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

Data Source

PatentUS11709894B2Task processing method and distributed computing framework

Publication Date: 2023.07.25 BEIJING BAIDU NETCOM SCI & TECH CO LTD

AI summary

The present disclosure discloses a task processing method and a distributed computing framework. A specific embodiment of the method includes: parsing an expression corresponding to a distributed computing task, and constructing task description information corresponding to the distributed computing task, the task description information being used to describe a corresponding relationship between an operator and a distributed dataset, and the operator acting on at least one of the distributed dataset or distributed datasets obtained by grouping the distributed dataset; determining, based on the task description information, a distributed dataset the operator acting on; and performing distributed computing on the distributed dataset the operator acting on using the operator. In the distributed computing, the acting scope and nesting relationship of the operator is described by constructing a topology.