Parallel Decision System for Distributed Data Processing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

In distributed data processing systems, manual adjustments for parallel decisions across thousands or millions of data processing nodes are labor-intensive and inefficient, leading to talent waste and suboptimal computation efficiency due to varying parallel processing modes and transmission overheads.

Innovation Solution

A parallel decision system that automatically determines the optimal parallel mode for each logical node by generating an initial logical node topology, traversing the topology to compute and transform configurations, and applying a local greedy strategy to minimize transmission costs, thereby reducing the solution space and computation costs.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If manual adjustments are made for parallel decisions across thousands or millions of data processing nodes, then parallel processing modes can be optimized, but labor intensity increases and talent waste occurs

Engineering Contradiction:

Improvecomputation efficiencyVSAvoidmanual adjustment complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The system automatically determines optimal parallel modes for each logical node by generating initial topologies, traversing them to compute configurations, and applying local greedy strategies to minimize transmission costs. This self-service mechanism eliminates manual adjustments while achieving optimized computation efficiency across distributed data processing nodes.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system transforms the complex parallel decision problem into a series of simpler parameter optimization problems by breaking down the topology into predetermined configurations (intermediate nodes, paired nodes, end nodes) and applying different cost computation strategies to each type, thereby reducing the complexity of manual adjustment requirements.

Inventive Principle:

Principle #35Parameter changes

2Power

If different parallel processing modes are applied in different logical nodes, then computing power limitations can be satisfied, but data transmission overhead increases and computation efficiency decreases

Engineering Contradiction:

Improvecomputing power utilizationVSAvoiddata transmission overhead

Core Design Contradiction:

PowerVSLoss of energy

Solution Approach 1:

The system applies different parallel processing modes to different logical nodes based on their specific roles and requirements. Each logical node is assigned an optimal parallel mode from candidate sets, allowing local optimization of computing power utilization while minimizing data transmission overhead through coordinated decision-making across the distributed topology.

Inventive Principle:

Principle #3Local quality

3Productivity

If manual repeated adjustments are made for parallel decisions, then parallel modes can be optimized, but time consumption increases and talent waste occurs

Engineering Contradiction:

Improvecomputation efficiencyVSAvoidadjustment time

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The system performs preliminary actions by automatically generating initial logical node topologies and pre-computing candidate parallel solutions with their associated costs before actual execution. This preliminary configuration eliminates the need for manual repeated adjustments during the computation process, significantly reducing adjustment time while maintaining optimized computation efficiency.

Inventive Principle:

Principle #10Preliminary action

4Adaptability or versatility

If thousands or millions of parallel decisions are made manually, then parallel processing can be configured, but workload increases and human power is consumed

Engineering Contradiction:

Improveparallel processing configurationVSAvoidoperation simplicity

Core Design Contradiction:

Adaptability or versatilityVSEase of operation

Solution Approach 1:

The system automatically configures parallel processing for thousands or millions of logical nodes by generating topologies, traversing configurations, and applying greedy strategies to determine optimal parallel modes. This self-service approach maintains high adaptability and versatility in parallel processing configuration while completely eliminating manual operation requirements.

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS11782724B2Parallel decision system and method for distributed data processing

Publication Date: 2023.10.10 BEIJING SILICONFLOW TECHNOLOGY CO LTD

AI summary

The present disclosure provides a parallel decision system and method for distributed data processing. The system includes: an initial logical node generation assembly, a logical node traversal assembly, a predetermined configuration cost computation assembly, and a parallel decision assembly. The initial logical node generation assembly is configured to receive task configuration data input by a user to generate an initial logical node topology for the distributed data processing system. The logical node traversal assembly is configured to traverse the initial logical node topology to obtain a predetermined configuration in the initial logical node topology. The predetermined configuration cost computation assembly is configured to compute a transmission cost of each predetermined configuration and a cost sum. The predetermined configuration transformation assembly is configured to, based on the result of the predetermined configuration, reduce an initial logical node, and a connection edge, a combined initial logical node.